Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datahub-ingestion): refactor datahub ingestion slim images (#8515) #122

Merged
merged 5 commits into from
Oct 2, 2023

use variable for action image

dba564b
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Merged

feat(datahub-ingestion): refactor datahub ingestion slim images (#8515) #122

use variable for action image
dba564b
Select commit
Loading
Failed to load commit list.
GitHub Actions / Unit Test Results (metadata ingestion) failed Oct 1, 2023 in 0s

7 fail, 6 skipped, 1 055 pass in 1h 27m 23s

       8 files  ±0         8 suites  ±0   1h 27m 23s ⏱️ + 10m 29s
1 068 tests ±0  1 055 ✔️  - 4    6 💤 ±0  7 ❌ +4 
2 136 runs  ±0  2 115 ✔️  - 4  13 💤 ±0  8 ❌ +4 

Results for commit dba564b. ± Comparison against earlier commit 5b70610.

Annotations

Check warning on line 0 in tests.integration.delta_lake.test_delta_lake_minio

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

All 2 runs failed: test_delta_lake_ingest (tests.integration.delta_lake.test_delta_lake_minio) 

artifacts/Test Results (metadata ingestion 3.10)/metadata-ingestion/junit.slow.integration.xml [took 32s]
artifacts/Test Results (metadata ingestion 3.7)/metadata-ingestion/junit.slow.integration.xml [took 32s]
Raw output 
OSError: Generic S3 error: response error "request error", after 0 retries: builder error for url (http://localhost:9000/my-test-bucket/delta_tables/sales/_delta_log/_last_checkpoint): URL scheme is not allowed
pytestconfig = <_pytest.config.Config object at 0x7fd022cb1e10>
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_delta_lake_ingest0')
mock_time = None

    @pytest.mark.slow_integration
    def test_delta_lake_ingest(pytestconfig, tmp_path, mock_time):
        test_resources_dir = pytestconfig.rootpath / "tests/integration/delta_lake/"
    
        # Run the metadata ingestion pipeline.
        pipeline = Pipeline.create(
            {
                "run_id": "delta-lake-test",
                "source": {
                    "type": "delta-lake",
                    "config": {
                        "env": "DEV",
                        "base_path": "s3://my-test-bucket/delta_tables/sales",
                        "s3": {
                            "aws_config": {
                                "aws_access_key_id": "minioadmin",
                                "aws_secret_access_key": "minioadmin",
                                "aws_endpoint_url": "http://localhost:9000",
                                "aws_region": "us-east-1",
                            },
                        },
                    },
                },
                "sink": {
                    "type": "file",
                    "config": {
                        "filename": f"{tmp_path}/delta_lake_minio_mces.json",
                    },
                },
            }
        )
>       pipeline.run()

tests/integration/delta_lake/test_delta_lake_minio.py:95: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/datahub/ingestion/run/pipeline.py:365: in run
    for wu in itertools.islice(
src/datahub/ingestion/api/source_helpers.py:133: in auto_workunit_reporter
    for wu in stream:
src/datahub/ingestion/source/delta_lake/source.py:353: in get_workunits_internal
    yield from self.process_folder(self.source_config.complete_path)
src/datahub/ingestion/source/delta_lake/source.py:312: in process_folder
    delta_table = read_delta_table(path, self.storage_options, self.source_config)
src/datahub/ingestion/source/delta_lake/delta_lake_utils.py:44: in read_delta_table
    raise e
src/datahub/ingestion/source/delta_lake/delta_lake_utils.py:31: in read_delta_table
    return DeltaTable(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = DeltaTable(), table_uri = 's3://my-test-bucket/delta_tables/sales'
version = None
storage_options = {'AWS_ACCESS_KEY_ID': 'minioadmin', 'AWS_ENDPOINT_URL': 'http://localhost:9000', 'AWS_REGION': 'us-east-1', 'AWS_SECRET_ACCESS_KEY': 'minioadmin', ...}
without_files = False, log_buffer_size = None

    def __init__(
        self,
        table_uri: Union[str, Path],
        version: Optional[int] = None,
        storage_options: Optional[Dict[str, str]] = None,
        without_files: bool = False,
        log_buffer_size: Optional[int] = None,
    ):
        """
        Create the Delta Table from a path with an optional version.
        Multiple StorageBackends are currently supported: AWS S3, Azure Data Lake Storage Gen2, Google Cloud Storage (GCS) and local URI.
        Depending on the storage backend used, you could provide options values using the ``storage_options`` parameter.
    
        :param table_uri: the path of the DeltaTable
        :param version: version of the DeltaTable
        :param storage_options: a dictionary of the options to use for the storage backend
        :param without_files: If True, will load table without tracking files.
                              Some append-only applications might have no need of tracking any files. So, the
                              DeltaTable will be loaded with a significant memory reduction.
        :param log_buffer_size: Number of files to buffer when reading the commit log. A positive integer.
                                Setting a value greater than 1 results in concurrent calls to the storage api.
                                This can decrease latency if there are many files in the log since the last checkpoint,
                                but will also increase memory usage. Possible rate limits of the storage backend should
                                also be considered for optimal performance. Defaults to 4 * number of cpus.
    
        """
        self._storage_options = storage_options
>       self._table = RawDeltaTable(
            str(table_uri),
            version=version,
            storage_options=storage_options,
            without_files=without_files,
            log_buffer_size=log_buffer_size,
        )
E       OSError: Generic S3 error: response error "request error", after 0 retries: builder error for url (http://localhost:9000/my-test-bucket/delta_tables/sales/_delta_log/_last_checkpoint): URL scheme is not allowed

venv/lib/python3.10/site-packages/deltalake/table.py:247: OSError

Check warning on line 0 in tests.integration.s3.test_s3

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_data_lake_s3_ingest[folder_no_partition_glob.json] (tests.integration.s3.test_s3) 

artifacts/Test Results (metadata ingestion 3.7)/metadata-ingestion/junit.integration.xml [took 2s]
Raw output 
AssertionError: MCEs differ
{'values_changed': {"root[0]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
                                                                                              'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
 output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}]
pytestconfig = <_pytest.config.Config object at 0x7fac0b361fd0>
s3_populate = None, source_file = 'folder_no_partition_glob.json'
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_data_lake_s3_ingest_folde1')
mock_time = None

    @pytest.mark.integration
    @pytest.mark.parametrize("source_file", source_files)
    def test_data_lake_s3_ingest(
        pytestconfig, s3_populate, source_file, tmp_path, mock_time
    ):
        test_resources_dir = pytestconfig.rootpath / "tests/integration/s3/"
    
        f = open(os.path.join(SOURCE_FILES_PATH, source_file))
        source = json.load(f)
    
        config_dict = {}
        config_dict["source"] = source
        config_dict["sink"] = {
            "type": "file",
            "config": {
                "filename": f"{tmp_path}/{source_file}",
            },
        }
    
        config_dict["run_id"] = source_file
    
        pipeline = Pipeline.create(config_dict)
        pipeline.run()
        pipeline.raise_from_status()
    
        # Verify the output.
        mce_helpers.check_golden_file(
            pytestconfig,
            output_path=f"{tmp_path}/{source_file}",
            golden_path=f"{test_resources_dir}/golden-files/s3/golden_mces_{source_file}",
            ignore_paths=[
>               r"root\[\d+\]\['aspect'\]\['json'\]\['lastUpdatedTimestamp'\]",
            ],
        )

tests/integration/s3/test_s3.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_helpers/mce_helpers.py:163: in check_golden_file
    raise e
tests/test_helpers/mce_helpers.py:154: in check_golden_file
    assert_mces_equal(output, golden, ignore_paths)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

output = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
golden = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
ignore_paths = ["root\\[\\d+\\]\\['aspect'\\]\\['json'\\]\\['lastUpdatedTimestamp'\\]"]

    def assert_mces_equal(
        output: object, golden: object, ignore_paths: Optional[List[str]] = None
    ) -> None:
        # This method assumes we're given a list of MCE json objects.
        diff = deepdiff.DeepDiff(
            golden, output, exclude_regex_paths=ignore_paths, ignore_order=True
        )
        if diff:
            # Attempt a clean diff (removing None-s)
            assert isinstance(output, list)
            assert isinstance(golden, list)
            clean_output = [clean_nones(o) for o in output]
            clean_golden = [clean_nones(g) for g in golden]
            clean_diff = deepdiff.DeepDiff(
                clean_golden,
                clean_output,
                exclude_regex_paths=ignore_paths,
                ignore_order=True,
            )
            if not clean_diff:
                logger.debug(f"MCE-s differ, clean MCE-s are fine\n{pprint.pformat(diff)}")
            diff = clean_diff
            if diff:
                # do some additional processing to emit helpful messages
                output_urns = _get_entity_urns(output)
                golden_urns = _get_entity_urns(golden)
                in_golden_but_not_in_output = golden_urns - output_urns
                in_output_but_not_in_golden = output_urns - golden_urns
                if in_golden_but_not_in_output:
                    logger.info(
                        f"Golden file has {len(in_golden_but_not_in_output)} more urns: {in_golden_but_not_in_output}"
                    )
                if in_output_but_not_in_golden:
                    logger.info(
                        f"Golden file has {len(in_output_but_not_in_golden)} more urns: {in_output_but_not_in_golden}"
                    )
    
        assert (
            not diff
>       ), f"MCEs differ\n{pprint.pformat(diff)} \n output was: {json.dumps(output)}"
E       AssertionError: MCEs differ
E       {'values_changed': {"root[0]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
E                                                                                                     'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
E        output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_glob.json"}}]

tests/test_helpers/mce_helpers.py:117: AssertionError

Check warning on line 0 in tests.integration.s3.test_s3

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_data_lake_s3_ingest[folder_no_partition.json] (tests.integration.s3.test_s3) 

artifacts/Test Results (metadata ingestion 3.7)/metadata-ingestion/junit.integration.xml [took 2s]
Raw output 
AssertionError: MCEs differ
{'values_changed': {"root[23]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
                                                                                               'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
 output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv/part1.csv"}, "name": "food_csv", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_csv", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "boolean", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200374000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}]
pytestconfig = <_pytest.config.Config object at 0x7fac0b361fd0>
s3_populate = None, source_file = 'folder_no_partition.json'
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_data_lake_s3_ingest_folde4')
mock_time = None

    @pytest.mark.integration
    @pytest.mark.parametrize("source_file", source_files)
    def test_data_lake_s3_ingest(
        pytestconfig, s3_populate, source_file, tmp_path, mock_time
    ):
        test_resources_dir = pytestconfig.rootpath / "tests/integration/s3/"
    
        f = open(os.path.join(SOURCE_FILES_PATH, source_file))
        source = json.load(f)
    
        config_dict = {}
        config_dict["source"] = source
        config_dict["sink"] = {
            "type": "file",
            "config": {
                "filename": f"{tmp_path}/{source_file}",
            },
        }
    
        config_dict["run_id"] = source_file
    
        pipeline = Pipeline.create(config_dict)
        pipeline.run()
        pipeline.raise_from_status()
    
        # Verify the output.
        mce_helpers.check_golden_file(
            pytestconfig,
            output_path=f"{tmp_path}/{source_file}",
            golden_path=f"{test_resources_dir}/golden-files/s3/golden_mces_{source_file}",
            ignore_paths=[
>               r"root\[\d+\]\['aspect'\]\['json'\]\['lastUpdatedTimestamp'\]",
            ],
        )

tests/integration/s3/test_s3.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_helpers/mce_helpers.py:163: in check_golden_file
    raise e
tests/test_helpers/mce_helpers.py:154: in check_golden_file
    assert_mces_equal(output, golden, ignore_paths)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

output = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
golden = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
ignore_paths = ["root\\[\\d+\\]\\['aspect'\\]\\['json'\\]\\['lastUpdatedTimestamp'\\]"]

    def assert_mces_equal(
        output: object, golden: object, ignore_paths: Optional[List[str]] = None
    ) -> None:
        # This method assumes we're given a list of MCE json objects.
        diff = deepdiff.DeepDiff(
            golden, output, exclude_regex_paths=ignore_paths, ignore_order=True
        )
        if diff:
            # Attempt a clean diff (removing None-s)
            assert isinstance(output, list)
            assert isinstance(golden, list)
            clean_output = [clean_nones(o) for o in output]
            clean_golden = [clean_nones(g) for g in golden]
            clean_diff = deepdiff.DeepDiff(
                clean_golden,
                clean_output,
                exclude_regex_paths=ignore_paths,
                ignore_order=True,
            )
            if not clean_diff:
                logger.debug(f"MCE-s differ, clean MCE-s are fine\n{pprint.pformat(diff)}")
            diff = clean_diff
            if diff:
                # do some additional processing to emit helpful messages
                output_urns = _get_entity_urns(output)
                golden_urns = _get_entity_urns(golden)
                in_golden_but_not_in_output = golden_urns - output_urns
                in_output_but_not_in_golden = output_urns - golden_urns
                if in_golden_but_not_in_output:
                    logger.info(
                        f"Golden file has {len(in_golden_but_not_in_output)} more urns: {in_golden_but_not_in_output}"
                    )
                if in_output_but_not_in_golden:
                    logger.info(
                        f"Golden file has {len(in_output_but_not_in_golden)} more urns: {in_output_but_not_in_golden}"
                    )
    
        assert (
            not diff
>       ), f"MCEs differ\n{pprint.pformat(diff)} \n output was: {json.dumps(output)}"
E       AssertionError: MCEs differ
E       {'values_changed': {"root[23]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
E                                                                                                      'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
E        output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv/part1.csv"}, "name": "food_csv", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_csv", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "boolean", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200374000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition.json"}}]

tests/test_helpers/mce_helpers.py:117: AssertionError

Check warning on line 0 in tests.integration.s3.test_s3

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_data_lake_s3_ingest[folder_no_partition_exclude.json] (tests.integration.s3.test_s3) 

artifacts/Test Results (metadata ingestion 3.7)/metadata-ingestion/junit.integration.xml [took 2s]
Raw output 
AssertionError: MCEs differ
{'values_changed': {"root[0]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
                                                                                              'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
 output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}]
pytestconfig = <_pytest.config.Config object at 0x7fac0b361fd0>
s3_populate = None, source_file = 'folder_no_partition_exclude.json'
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_data_lake_s3_ingest_folde5')
mock_time = None

    @pytest.mark.integration
    @pytest.mark.parametrize("source_file", source_files)
    def test_data_lake_s3_ingest(
        pytestconfig, s3_populate, source_file, tmp_path, mock_time
    ):
        test_resources_dir = pytestconfig.rootpath / "tests/integration/s3/"
    
        f = open(os.path.join(SOURCE_FILES_PATH, source_file))
        source = json.load(f)
    
        config_dict = {}
        config_dict["source"] = source
        config_dict["sink"] = {
            "type": "file",
            "config": {
                "filename": f"{tmp_path}/{source_file}",
            },
        }
    
        config_dict["run_id"] = source_file
    
        pipeline = Pipeline.create(config_dict)
        pipeline.run()
        pipeline.raise_from_status()
    
        # Verify the output.
        mce_helpers.check_golden_file(
            pytestconfig,
            output_path=f"{tmp_path}/{source_file}",
            golden_path=f"{test_resources_dir}/golden-files/s3/golden_mces_{source_file}",
            ignore_paths=[
>               r"root\[\d+\]\['aspect'\]\['json'\]\['lastUpdatedTimestamp'\]",
            ],
        )

tests/integration/s3/test_s3.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_helpers/mce_helpers.py:163: in check_golden_file
    raise e
tests/test_helpers/mce_helpers.py:154: in check_golden_file
    assert_mces_equal(output, golden, ignore_paths)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

output = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
golden = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
ignore_paths = ["root\\[\\d+\\]\\['aspect'\\]\\['json'\\]\\['lastUpdatedTimestamp'\\]"]

    def assert_mces_equal(
        output: object, golden: object, ignore_paths: Optional[List[str]] = None
    ) -> None:
        # This method assumes we're given a list of MCE json objects.
        diff = deepdiff.DeepDiff(
            golden, output, exclude_regex_paths=ignore_paths, ignore_order=True
        )
        if diff:
            # Attempt a clean diff (removing None-s)
            assert isinstance(output, list)
            assert isinstance(golden, list)
            clean_output = [clean_nones(o) for o in output]
            clean_golden = [clean_nones(g) for g in golden]
            clean_diff = deepdiff.DeepDiff(
                clean_golden,
                clean_output,
                exclude_regex_paths=ignore_paths,
                ignore_order=True,
            )
            if not clean_diff:
                logger.debug(f"MCE-s differ, clean MCE-s are fine\n{pprint.pformat(diff)}")
            diff = clean_diff
            if diff:
                # do some additional processing to emit helpful messages
                output_urns = _get_entity_urns(output)
                golden_urns = _get_entity_urns(golden)
                in_golden_but_not_in_output = golden_urns - output_urns
                in_output_but_not_in_golden = output_urns - golden_urns
                if in_golden_but_not_in_output:
                    logger.info(
                        f"Golden file has {len(in_golden_but_not_in_output)} more urns: {in_golden_but_not_in_output}"
                    )
                if in_output_but_not_in_golden:
                    logger.info(
                        f"Golden file has {len(in_output_but_not_in_golden)} more urns: {in_output_but_not_in_golden}"
                    )
    
        assert (
            not diff
>       ), f"MCEs differ\n{pprint.pformat(diff)} \n output was: {json.dumps(output)}"
E       AssertionError: MCEs differ
E       {'values_changed': {"root[0]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
E                                                                                                     'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
E        output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_exclude.json"}}]

tests/test_helpers/mce_helpers.py:117: AssertionError

Check warning on line 0 in tests.integration.s3.test_s3

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_data_lake_s3_ingest[folder_no_partition_filename.json] (tests.integration.s3.test_s3) 

artifacts/Test Results (metadata ingestion 3.7)/metadata-ingestion/junit.integration.xml [took 2s]
Raw output 
AssertionError: MCEs differ
{'values_changed': {"root[23]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
                                                                                               'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
 output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv/part1.csv"}, "name": "folder_aaa.food_csv", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "folder_aaa.food_csv", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "boolean", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200374000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "folder_aaa.food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "folder_aaa.food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}]
pytestconfig = <_pytest.config.Config object at 0x7fac0b361fd0>
s3_populate = None, source_file = 'folder_no_partition_filename.json'
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/test_data_lake_s3_ingest_folde6')
mock_time = None

    @pytest.mark.integration
    @pytest.mark.parametrize("source_file", source_files)
    def test_data_lake_s3_ingest(
        pytestconfig, s3_populate, source_file, tmp_path, mock_time
    ):
        test_resources_dir = pytestconfig.rootpath / "tests/integration/s3/"
    
        f = open(os.path.join(SOURCE_FILES_PATH, source_file))
        source = json.load(f)
    
        config_dict = {}
        config_dict["source"] = source
        config_dict["sink"] = {
            "type": "file",
            "config": {
                "filename": f"{tmp_path}/{source_file}",
            },
        }
    
        config_dict["run_id"] = source_file
    
        pipeline = Pipeline.create(config_dict)
        pipeline.run()
        pipeline.raise_from_status()
    
        # Verify the output.
        mce_helpers.check_golden_file(
            pytestconfig,
            output_path=f"{tmp_path}/{source_file}",
            golden_path=f"{test_resources_dir}/golden-files/s3/golden_mces_{source_file}",
            ignore_paths=[
>               r"root\[\d+\]\['aspect'\]\['json'\]\['lastUpdatedTimestamp'\]",
            ],
        )

tests/integration/s3/test_s3.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_helpers/mce_helpers.py:163: in check_golden_file
    raise e
tests/test_helpers/mce_helpers.py:154: in check_golden_file
    assert_mces_equal(output, golden, ignore_paths)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

output = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
golden = [{'aspect': {'json': {'customProperties': {'schema_inferred_from': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/...dataPlatform:s3'}}, 'aspectName': 'dataPlatformInstance', 'changeType': 'UPSERT', 'entityType': 'container', ...}, ...]
ignore_paths = ["root\\[\\d+\\]\\['aspect'\\]\\['json'\\]\\['lastUpdatedTimestamp'\\]"]

    def assert_mces_equal(
        output: object, golden: object, ignore_paths: Optional[List[str]] = None
    ) -> None:
        # This method assumes we're given a list of MCE json objects.
        diff = deepdiff.DeepDiff(
            golden, output, exclude_regex_paths=ignore_paths, ignore_order=True
        )
        if diff:
            # Attempt a clean diff (removing None-s)
            assert isinstance(output, list)
            assert isinstance(golden, list)
            clean_output = [clean_nones(o) for o in output]
            clean_golden = [clean_nones(g) for g in golden]
            clean_diff = deepdiff.DeepDiff(
                clean_golden,
                clean_output,
                exclude_regex_paths=ignore_paths,
                ignore_order=True,
            )
            if not clean_diff:
                logger.debug(f"MCE-s differ, clean MCE-s are fine\n{pprint.pformat(diff)}")
            diff = clean_diff
            if diff:
                # do some additional processing to emit helpful messages
                output_urns = _get_entity_urns(output)
                golden_urns = _get_entity_urns(golden)
                in_golden_but_not_in_output = golden_urns - output_urns
                in_output_but_not_in_golden = output_urns - golden_urns
                if in_golden_but_not_in_output:
                    logger.info(
                        f"Golden file has {len(in_golden_but_not_in_output)} more urns: {in_golden_but_not_in_output}"
                    )
                if in_output_but_not_in_golden:
                    logger.info(
                        f"Golden file has {len(in_output_but_not_in_golden)} more urns: {in_output_but_not_in_golden}"
                    )
    
        assert (
            not diff
>       ), f"MCEs differ\n{pprint.pformat(diff)} \n output was: {json.dumps(output)}"
E       AssertionError: MCEs differ
E       {'values_changed': {"root[23]['aspect']['json']['customProperties']['schema_inferred_from']": {'new_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet',
E                                                                                                      'old_value': 's3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part1.parquet'}}} 
E        output was: [{"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv/part1.csv"}, "name": "folder_aaa.food_csv", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "folder_aaa.food_csv", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "boolean", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "integer", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200374000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "bucket_name": "my-test-bucket"}, "name": "my-test-bucket"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:050fedde7a12cb8c8447db8d298f5577", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["S3 bucket"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a"}, "name": "folder_a"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:86297df39321e4948dbe8b8e941de98b", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:050fedde7a12cb8c8447db8d298f5577"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa"}, "name": "folder_aa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:273fbeff7bd9ecb74982205aadd77994", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:86297df39321e4948dbe8b8e941de98b"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "containerProperties", "aspect": {"json": {"customProperties": {"platform": "s3", "env": "UAT", "folder_abs_path": "my-test-bucket/folder_a/folder_aa/folder_aaa"}, "name": "folder_aaa"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"json": {"platform": "urn:li:dataPlatform:s3"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "subTypes", "aspect": {"json": {"typeNames": ["Folder"]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "container", "entityUrn": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:273fbeff7bd9ecb74982205aadd77994"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "datasetProperties", "aspect": {"json": {"customProperties": {"schema_inferred_from": "s3://my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet/part2.parquet"}, "name": "folder_aaa.food_parquet", "description": "", "tags": []}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "schemaMetadata", "aspect": {"json": {"schemaName": "folder_aaa.food_parquet", "platform": "urn:li:dataPlatform:s3", "version": 0, "created": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "lastModified": {"time": 0, "actor": "urn:li:corpuser:unknown"}, "hash": "", "platformSchema": {"com.linkedin.schema.OtherSchema": {"rawSchema": ""}}, "fields": [{"fieldPath": "color", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "healthy", "nullable": false, "type": {"type": {"com.linkedin.schema.BooleanType": {}}}, "nativeDataType": "bool", "recursive": false, "isPartOfKey": false}, {"fieldPath": "height", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}, {"fieldPath": "name", "nullable": false, "type": {"type": {"com.linkedin.schema.StringType": {}}}, "nativeDataType": "string", "recursive": false, "isPartOfKey": false}, {"fieldPath": "weight", "nullable": false, "type": {"type": {"com.linkedin.schema.NumberType": {}}}, "nativeDataType": "int64", "recursive": false, "isPartOfKey": false}]}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "operation", "aspect": {"json": {"timestampMillis": 1615443388097, "partitionSpec": {"type": "FULL_TABLE", "partition": "FULL_TABLE_SNAPSHOT"}, "operationType": "UPDATE", "lastUpdatedTimestamp": 1696200375000}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "container", "aspect": {"json": {"container": "urn:li:container:ec362903c4c7de60197fcc7b7a79e4c2"}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_csv,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}, {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-test-bucket/folder_a/folder_aa/folder_aaa/food_parquet,UAT)", "changeType": "UPSERT", "aspectName": "status", "aspect": {"json": {"removed": false}}, "systemMetadata": {"lastObserved": 1615443388097, "runId": "folder_no_partition_filename.json"}}]

tests/test_helpers/mce_helpers.py:117: AssertionError

Check warning on line 0 in tests.unit.test_airflow

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_lineage_backend[airflow-lineage-no-executions] (tests.unit.test_airflow) 

artifacts/Test Results (metadata ingestion 3.10)/metadata-ingestion/junit.quick.xml [took 0s]
Raw output 
ValueError: The run_id provided 'scheduled_2020-05-17 00:00:00' does not match the pattern '^[A-Za-z0-9_.~:+-]+$' or '^(?:manual|scheduled|dataset_triggered)__(?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+00:00)$'
mock_emit = <MagicMock name='make_emitter' id='139894779178400'>
inlets = [Dataset(platform='snowflake', name='mydb.schema.tableConsumed', env='PROD')]
outlets = [Dataset(platform='snowflake', name='mydb.schema.tableProduced', env='PROD')]
capture_executions = False

    @pytest.mark.parametrize(
        ["inlets", "outlets", "capture_executions"],
        [
            pytest.param(
                [Dataset("snowflake", "mydb.schema.tableConsumed")],
                [Dataset("snowflake", "mydb.schema.tableProduced")],
                False,
                id="airflow-lineage-no-executions",
            ),
            pytest.param(
                [Dataset("snowflake", "mydb.schema.tableConsumed")],
                [Dataset("snowflake", "mydb.schema.tableProduced")],
                True,
                id="airflow-lineage-capture-executions",
            ),
        ],
    )
    @mock.patch("datahub_provider.hooks.datahub.DatahubRestHook.make_emitter")
    def test_lineage_backend(mock_emit, inlets, outlets, capture_executions):
        DEFAULT_DATE = datetime.datetime(2020, 5, 17)
        mock_emitter = Mock()
        mock_emit.return_value = mock_emitter
        # Using autospec on xcom_pull and xcom_push methods fails on Python 3.6.
        with mock.patch.dict(
            os.environ,
            {
                "AIRFLOW__LINEAGE__BACKEND": "datahub_provider.lineage.datahub.DatahubLineageBackend",
                "AIRFLOW__LINEAGE__DATAHUB_CONN_ID": datahub_rest_connection_config.conn_id,
                "AIRFLOW__LINEAGE__DATAHUB_KWARGS": json.dumps(
                    {"graceful_exceptions": False, "capture_executions": capture_executions}
                ),
            },
        ), mock.patch("airflow.models.BaseOperator.xcom_pull"), mock.patch(
            "airflow.models.BaseOperator.xcom_push"
        ), patch_airflow_connection(
            datahub_rest_connection_config
        ):
            func = mock.Mock()
            func.__name__ = "foo"
    
            dag = DAG(dag_id="test_lineage_is_sent_to_backend", start_date=DEFAULT_DATE)
    
            with dag:
                op1 = DummyOperator(
                    task_id="task1_upstream",
                    inlets=inlets,
                    outlets=outlets,
                )
                op2 = DummyOperator(
                    task_id="task2",
                    inlets=inlets,
                    outlets=outlets,
                )
                op1 >> op2
    
            # Airflow < 2.2 requires the execution_date parameter. Newer Airflow
            # versions do not require it, but will attempt to find the associated
            # run_id in the database if execution_date is provided. As such, we
            # must fake the run_id parameter for newer Airflow versions.
            if AIRFLOW_VERSION < packaging.version.parse("2.2.0"):
                ti = TaskInstance(task=op2, execution_date=DEFAULT_DATE)
                # Ignoring type here because DagRun state is just a sring at Airflow 1
                dag_run = DagRun(state="success", run_id=f"scheduled_{DEFAULT_DATE}")  # type: ignore
            else:
                from airflow.utils.state import DagRunState
    
                ti = TaskInstance(task=op2, run_id=f"test_airflow-{DEFAULT_DATE}")
>               dag_run = DagRun(
                    state=DagRunState.SUCCESS, run_id=f"scheduled_{DEFAULT_DATE}"
                )

tests/unit/test_airflow.py:265: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
<string>:4: in __init__
    ???
venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py:481: in _initialize_instance
    with util.safe_reraise():
venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py:70: in __exit__
    compat.raise_(
venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py:208: in raise_
    raise exception
venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py:479: in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
venv/lib/python3.10/site-packages/airflow/models/dagrun.py:217: in __init__
    self.run_id = run_id
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:460: in __set__
    self.impl.set(
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:1098: in set
    value = self.fire_replace_event(
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:1106: in fire_replace_event
    value = fn(
venv/lib/python3.10/site-packages/sqlalchemy/orm/util.py:185: in set_
    return validator(state.obj(), key, value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <DagRun None @ None: None, state:None, queued_at: None. externally triggered: None>
key = 'run_id', run_id = 'scheduled_2020-05-17 00:00:00'

    @validates("run_id")
    def validate_run_id(self, key: str, run_id: str) -> str | None:
        if not run_id:
            return None
        regex = airflow_conf.get("scheduler", "allowed_run_id_pattern")
        if not re2.match(regex, run_id) and not re2.match(RUN_ID_REGEX, run_id):
>           raise ValueError(
                f"The run_id provided '{run_id}' does not match the pattern '{regex}' or '{RUN_ID_REGEX}'"
            )
E           ValueError: The run_id provided 'scheduled_2020-05-17 00:00:00' does not match the pattern '^[A-Za-z0-9_.~:+-]+$' or '^(?:manual|scheduled|dataset_triggered)__(?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+00:00)$'

venv/lib/python3.10/site-packages/airflow/models/dagrun.py:245: ValueError

Check warning on line 0 in tests.unit.test_airflow

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1 out of 2 runs failed: test_lineage_backend[airflow-lineage-capture-executions] (tests.unit.test_airflow) 

artifacts/Test Results (metadata ingestion 3.10)/metadata-ingestion/junit.quick.xml [took 0s]
Raw output 
ValueError: The run_id provided 'scheduled_2020-05-17 00:00:00' does not match the pattern '^[A-Za-z0-9_.~:+-]+$' or '^(?:manual|scheduled|dataset_triggered)__(?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+00:00)$'
mock_emit = <MagicMock name='make_emitter' id='139894775205792'>
inlets = [Dataset(platform='snowflake', name='mydb.schema.tableConsumed', env='PROD')]
outlets = [Dataset(platform='snowflake', name='mydb.schema.tableProduced', env='PROD')]
capture_executions = True

    @pytest.mark.parametrize(
        ["inlets", "outlets", "capture_executions"],
        [
            pytest.param(
                [Dataset("snowflake", "mydb.schema.tableConsumed")],
                [Dataset("snowflake", "mydb.schema.tableProduced")],
                False,
                id="airflow-lineage-no-executions",
            ),
            pytest.param(
                [Dataset("snowflake", "mydb.schema.tableConsumed")],
                [Dataset("snowflake", "mydb.schema.tableProduced")],
                True,
                id="airflow-lineage-capture-executions",
            ),
        ],
    )
    @mock.patch("datahub_provider.hooks.datahub.DatahubRestHook.make_emitter")
    def test_lineage_backend(mock_emit, inlets, outlets, capture_executions):
        DEFAULT_DATE = datetime.datetime(2020, 5, 17)
        mock_emitter = Mock()
        mock_emit.return_value = mock_emitter
        # Using autospec on xcom_pull and xcom_push methods fails on Python 3.6.
        with mock.patch.dict(
            os.environ,
            {
                "AIRFLOW__LINEAGE__BACKEND": "datahub_provider.lineage.datahub.DatahubLineageBackend",
                "AIRFLOW__LINEAGE__DATAHUB_CONN_ID": datahub_rest_connection_config.conn_id,
                "AIRFLOW__LINEAGE__DATAHUB_KWARGS": json.dumps(
                    {"graceful_exceptions": False, "capture_executions": capture_executions}
                ),
            },
        ), mock.patch("airflow.models.BaseOperator.xcom_pull"), mock.patch(
            "airflow.models.BaseOperator.xcom_push"
        ), patch_airflow_connection(
            datahub_rest_connection_config
        ):
            func = mock.Mock()
            func.__name__ = "foo"
    
            dag = DAG(dag_id="test_lineage_is_sent_to_backend", start_date=DEFAULT_DATE)
    
            with dag:
                op1 = DummyOperator(
                    task_id="task1_upstream",
                    inlets=inlets,
                    outlets=outlets,
                )
                op2 = DummyOperator(
                    task_id="task2",
                    inlets=inlets,
                    outlets=outlets,
                )
                op1 >> op2
    
            # Airflow < 2.2 requires the execution_date parameter. Newer Airflow
            # versions do not require it, but will attempt to find the associated
            # run_id in the database if execution_date is provided. As such, we
            # must fake the run_id parameter for newer Airflow versions.
            if AIRFLOW_VERSION < packaging.version.parse("2.2.0"):
                ti = TaskInstance(task=op2, execution_date=DEFAULT_DATE)
                # Ignoring type here because DagRun state is just a sring at Airflow 1
                dag_run = DagRun(state="success", run_id=f"scheduled_{DEFAULT_DATE}")  # type: ignore
            else:
                from airflow.utils.state import DagRunState
    
                ti = TaskInstance(task=op2, run_id=f"test_airflow-{DEFAULT_DATE}")
>               dag_run = DagRun(
                    state=DagRunState.SUCCESS, run_id=f"scheduled_{DEFAULT_DATE}"
                )

tests/unit/test_airflow.py:265: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
<string>:4: in __init__
    ???
venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py:481: in _initialize_instance
    with util.safe_reraise():
venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py:70: in __exit__
    compat.raise_(
venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py:208: in raise_
    raise exception
venv/lib/python3.10/site-packages/sqlalchemy/orm/state.py:479: in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
venv/lib/python3.10/site-packages/airflow/models/dagrun.py:217: in __init__
    self.run_id = run_id
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:460: in __set__
    self.impl.set(
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:1098: in set
    value = self.fire_replace_event(
venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py:1106: in fire_replace_event
    value = fn(
venv/lib/python3.10/site-packages/sqlalchemy/orm/util.py:185: in set_
    return validator(state.obj(), key, value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <DagRun None @ None: None, state:None, queued_at: None. externally triggered: None>
key = 'run_id', run_id = 'scheduled_2020-05-17 00:00:00'

    @validates("run_id")
    def validate_run_id(self, key: str, run_id: str) -> str | None:
        if not run_id:
            return None
        regex = airflow_conf.get("scheduler", "allowed_run_id_pattern")
        if not re2.match(regex, run_id) and not re2.match(RUN_ID_REGEX, run_id):
>           raise ValueError(
                f"The run_id provided '{run_id}' does not match the pattern '{regex}' or '{RUN_ID_REGEX}'"
            )
E           ValueError: The run_id provided 'scheduled_2020-05-17 00:00:00' does not match the pattern '^[A-Za-z0-9_.~:+-]+$' or '^(?:manual|scheduled|dataset_triggered)__(?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+00:00)$'

venv/lib/python3.10/site-packages/airflow/models/dagrun.py:245: ValueError

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

6 skipped tests found 

There are 6 skipped tests, see "Raw output" for the full list of skipped tests.
Raw output 
tests.integration.git.test_git_clone ‑ test_git_clone_private
tests.integration.hana.test_hana ‑ test_hana_ingest
tests.integration.vertica.test_vertica ‑ test_vertica_ingest_with_db
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_with_parenthesis_in_from
tests.unit.test_plugin_system ‑ test_list_all[False]
tests.unit.test_plugin_system ‑ test_list_all[True]

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1068 tests found (test 1 to 756) 

There are 1068 tests, see "Raw output" for the list of tests 1 to 756.
Raw output 
tests.integration.azure_ad.test_azure_ad ‑ test_azure_ad_config
tests.integration.azure_ad.test_azure_ad ‑ test_azure_ad_source_default_configs
tests.integration.azure_ad.test_azure_ad ‑ test_azure_ad_source_empty_group_membership
tests.integration.azure_ad.test_azure_ad ‑ test_azure_ad_source_nested_groups
tests.integration.azure_ad.test_azure_ad ‑ test_azure_ad_stateful_ingestion
tests.integration.azure_ad.test_azure_ad ‑ test_azure_source_ingestion_disabled
tests.integration.business-glossary.test_business_glossary ‑ test_auto_id_creation_on_reserved_char
tests.integration.business-glossary.test_business_glossary ‑ test_glossary_ingest[False-glossary_events_golden.json]
tests.integration.business-glossary.test_business_glossary ‑ test_glossary_ingest[True-glossary_events_auto_id_golden.json]
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_assertion_circuit_breaker_assertion_with_active_assertion
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_assertion_circuit_breaker_updated_at_after_last_assertion
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_assertion_circuit_breaker_with_empty_response
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_assertion_circuit_breaker_with_no_error
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_operation_circuit_breaker_with_empty_response
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_operation_circuit_breaker_with_not_recent_operation
tests.integration.circuit_breaker.test_circuit_breaker ‑ test_operation_circuit_breaker_with_valid_response
tests.integration.clickhouse.test_clickhouse ‑ test_clickhouse_ingest
tests.integration.csv-enricher.test_csv_enricher ‑ test_csv_enricher_config
tests.integration.csv-enricher.test_csv_enricher ‑ test_csv_enricher_source
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-column-meta-mapping]
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-test-with-complex-owner-patterns]
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-test-with-data-platform-instance]
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-test-with-non-incremental-lineage]
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-test-with-schemas-dbt-enabled]
tests.integration.dbt.test_dbt ‑ test_dbt_ingest[dbt-test-with-target-platform-instance]
tests.integration.dbt.test_dbt ‑ test_dbt_only_test_definitions_and_results
tests.integration.dbt.test_dbt ‑ test_dbt_tests
tests.integration.dbt.test_dbt ‑ test_dbt_tests_only_assertions
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[array(row(x bigint, y double))-array]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[bigint-bigint]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[boolean-boolean]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[char-char]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[date-date]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[decimal(10,0)-decimal]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[double-double]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[int-int]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[integer-integer]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[json-json]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[map(varchar, varchar)-map]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[real-real]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[row(x bigint, y double)-row]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[smallint-smallint]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[time(12)-time]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[time-time]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[timestamp(3)-timestamp]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[timestamp-timestamp]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[tinyint-tinyint]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[varbinary-varbinary]
tests.integration.dbt.test_dbt ‑ test_resolve_trino_modified_type[varchar(20)-varchar]
tests.integration.delta_lake.test_delta_lake_minio ‑ test_delta_lake_ingest
tests.integration.delta_lake.test_local_delta_lake ‑ test_delta_lake[allow_table.json]
tests.integration.delta_lake.test_local_delta_lake ‑ test_delta_lake[inner_table.json]
tests.integration.delta_lake.test_local_delta_lake ‑ test_delta_lake[relative_path.json]
tests.integration.delta_lake.test_local_delta_lake ‑ test_delta_lake[single_table.json]
tests.integration.delta_lake.test_local_delta_lake ‑ test_delta_lake_incorrect_config_raises_error
tests.integration.feast.test_feast_repository ‑ test_feast_repository_ingest
tests.integration.git.test_git_clone ‑ test_base_url_guessing
tests.integration.git.test_git_clone ‑ test_git_clone_private
tests.integration.git.test_git_clone ‑ test_git_clone_public
tests.integration.git.test_git_clone ‑ test_github_branch
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint-ge_mcps_golden.json]
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint_2-ge_mcps_golden_2.json]
tests.integration.hana.test_hana ‑ test_hana_ingest
tests.integration.hive.test_hive ‑ test_hive_ingest
tests.integration.hive.test_hive ‑ test_hive_ingest_all_db
tests.integration.hive.test_hive ‑ test_hive_instance_check
tests.integration.iceberg.test_iceberg ‑ test_iceberg_ingest
tests.integration.iceberg.test_iceberg ‑ test_iceberg_profiling
tests.integration.iceberg.test_iceberg ‑ test_iceberg_stateful_ingest
tests.integration.kafka-connect.test_kafka_connect ‑ test_kafka_connect_ingest
tests.integration.kafka-connect.test_kafka_connect ‑ test_kafka_connect_ingest_stateful
tests.integration.kafka-connect.test_kafka_connect ‑ test_kafka_connect_mongosourceconnect_ingest
tests.integration.kafka.test_kafka ‑ test_kafka_ingest
tests.integration.kafka.test_kafka_state ‑ test_kafka_ingest_with_stateful
tests.integration.ldap.test_ldap ‑ test_ldap_ingest
tests.integration.ldap.test_ldap ‑ test_ldap_memberof_ingest
tests.integration.ldap.test_ldap_stateful ‑ test_ldap_stateful
tests.integration.looker.test_looker ‑ test_looker_ingest
tests.integration.looker.test_looker ‑ test_looker_ingest_allow_pattern
tests.integration.looker.test_looker ‑ test_looker_ingest_joins
tests.integration.looker.test_looker ‑ test_looker_ingest_stateful
tests.integration.looker.test_looker ‑ test_looker_ingest_unaliased_joins
tests.integration.looker.test_looker ‑ test_looker_ingest_usage_history
tests.integration.lookml.test_lookml ‑ test_hive_platform_drops_ids
tests.integration.lookml.test_lookml ‑ test_lookml_bad_sql_parser
tests.integration.lookml.test_lookml ‑ test_lookml_base_folder
tests.integration.lookml.test_lookml ‑ test_lookml_explore_refinement
tests.integration.lookml.test_lookml ‑ test_lookml_git_info
tests.integration.lookml.test_lookml ‑ test_lookml_ingest
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_api_bigquery
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_api_hive
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_offline
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_offline_platform_instance
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_offline_with_model_deny
tests.integration.lookml.test_lookml ‑ test_lookml_ingest_stateful
tests.integration.lookml.test_lookml ‑ test_lookml_refinement_include_order
tests.integration.lookml.test_lookml ‑ test_lookml_refinement_ingest
tests.integration.lookml.test_lookml ‑ test_lookml_view_merge
tests.integration.lookml.test_lookml ‑ test_reachable_views
tests.integration.metabase.test_metabase ‑ test_mode_ingest_failure
tests.integration.metabase.test_metabase ‑ test_mode_ingest_success
tests.integration.mode.test_mode ‑ test_mode_ingest_failure
tests.integration.mode.test_mode ‑ test_mode_ingest_success
tests.integration.mongodb.test_mongodb ‑ test_mongodb_ingest
tests.integration.mysql.test_mysql ‑ test_mysql_ingest_no_db
tests.integration.mysql.test_mysql ‑ test_mysql_ingest_with_db
tests.integration.mysql.test_mysql ‑ test_mysql_ingest_with_db_alias
tests.integration.nifi.test_nifi ‑ test_nifi_ingest
tests.integration.okta.test_okta ‑ test_okta_config
tests.integration.okta.test_okta ‑ test_okta_source_custom_user_name_regex
tests.integration.okta.test_okta ‑ test_okta_source_default_configs
tests.integration.okta.test_okta ‑ test_okta_source_include_deprovisioned_suspended_users
tests.integration.okta.test_okta ‑ test_okta_source_ingestion_disabled
tests.integration.okta.test_okta ‑ test_okta_stateful_ingestion
tests.integration.openapi.test_openapi ‑ test_openapi_ingest
tests.integration.oracle.test_oracle ‑ test_oracle_source_integration_with_database
tests.integration.oracle.test_oracle ‑ test_oracle_source_integration_with_out_database
tests.integration.postgres.test_postgres ‑ test_postgres_ingest_with_all_db
tests.integration.postgres.test_postgres ‑ test_postgres_ingest_with_db
tests.integration.powerbi.test_admin_only_api ‑ test_admin_only_apis
tests.integration.powerbi.test_admin_only_api ‑ test_most_config_and_modified_since
tests.integration.powerbi.test_m_parser ‑ test_databricks_regular_case
tests.integration.powerbi.test_m_parser ‑ test_expression_is_none
tests.integration.powerbi.test_m_parser ‑ test_for_each_expression_1
tests.integration.powerbi.test_m_parser ‑ test_for_each_expression_2
tests.integration.powerbi.test_m_parser ‑ test_google_bigquery_1
tests.integration.powerbi.test_m_parser ‑ test_google_bigquery_2
tests.integration.powerbi.test_m_parser ‑ test_mssql_regular_case
tests.integration.powerbi.test_m_parser ‑ test_mssql_with_query
tests.integration.powerbi.test_m_parser ‑ test_multi_source_table
tests.integration.powerbi.test_m_parser ‑ test_native_query_disabled
tests.integration.powerbi.test_m_parser ‑ test_oracle_regular_case
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query1
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query10
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query11
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query12
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query13
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query2
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query3
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query4
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query5
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query6
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query7
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query8
tests.integration.powerbi.test_m_parser ‑ test_parse_m_query9
tests.integration.powerbi.test_m_parser ‑ test_postgres_regular_case
tests.integration.powerbi.test_m_parser ‑ test_redshift_native_query
tests.integration.powerbi.test_m_parser ‑ test_redshift_regular_case
tests.integration.powerbi.test_m_parser ‑ test_snowflake_native_query
tests.integration.powerbi.test_m_parser ‑ test_snowflake_regular_case
tests.integration.powerbi.test_m_parser ‑ test_table_combine
tests.integration.powerbi.test_native_sql_parser ‑ test_join
tests.integration.powerbi.test_native_sql_parser ‑ test_simple_from
tests.integration.powerbi.test_powerbi ‑ test_admin_access_is_not_allowed
tests.integration.powerbi.test_powerbi ‑ test_dataset_type_mapping_error
tests.integration.powerbi.test_powerbi ‑ test_dataset_type_mapping_should_set_to_all
tests.integration.powerbi.test_powerbi ‑ test_extract_endorsements
tests.integration.powerbi.test_powerbi ‑ test_extract_lineage
tests.integration.powerbi.test_powerbi ‑ test_extract_reports
tests.integration.powerbi.test_powerbi ‑ test_override_ownership
tests.integration.powerbi.test_powerbi ‑ test_powerbi_ingest
tests.integration.powerbi.test_powerbi ‑ test_powerbi_ingest_urn_lower_case
tests.integration.powerbi.test_powerbi ‑ test_powerbi_platform_instance_ingest
tests.integration.powerbi.test_powerbi ‑ test_reports_with_failed_page_request
tests.integration.powerbi.test_powerbi ‑ test_scan_all_workspaces
tests.integration.powerbi.test_powerbi ‑ test_server_to_platform_map
tests.integration.powerbi.test_powerbi ‑ test_workspace_container
tests.integration.powerbi.test_stateful_ingestion ‑ test_powerbi_stateful_ingestion
tests.integration.powerbi_report_server.test_powerbi_report_server ‑ test_powerbi_ingest
tests.integration.presto-on-hive.test_presto_on_hive ‑ test_presto_on_hive_ingest[hive-False-False-False-_1]
tests.integration.presto-on-hive.test_presto_on_hive ‑ test_presto_on_hive_ingest[hive-False-False-True-_3]
tests.integration.presto-on-hive.test_presto_on_hive ‑ test_presto_on_hive_ingest[presto-on-hive-True-True-False-_2]
tests.integration.presto-on-hive.test_presto_on_hive ‑ test_presto_on_hive_ingest[presto-on-hive-True-True-True-_4]
tests.integration.presto-on-hive.test_presto_on_hive ‑ test_presto_on_hive_instance_ingest
tests.integration.redshift-usage.test_redshift_usage ‑ test_redshift_usage_config
tests.integration.redshift-usage.test_redshift_usage ‑ test_redshift_usage_filtering
tests.integration.redshift-usage.test_redshift_usage ‑ test_redshift_usage_source
tests.integration.remote.test_remote ‑ test_remote_ingest
tests.integration.s3.test_s3 ‑ test_data_lake_incorrect_config_raises_error
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[file_without_extension.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_no_partition.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_no_partition_exclude.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_no_partition_filename.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_no_partition_glob.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_partition_basic.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_partition_keyval.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[folder_partition_update_schema.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[multiple_files.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[multiple_spec_for_files.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[multiple_specs_of_different_buckets.json]
tests.integration.s3.test_s3 ‑ test_data_lake_local_ingest[single_file.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[file_without_extension.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_no_partition.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_no_partition_exclude.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_no_partition_filename.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_no_partition_glob.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_partition_basic.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_partition_keyval.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[folder_partition_update_schema.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[multiple_files.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[multiple_spec_for_files.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[multiple_specs_of_different_buckets.json]
tests.integration.s3.test_s3 ‑ test_data_lake_s3_ingest[single_file.json]
tests.integration.salesforce.test_salesforce ‑ test_salesforce_ingest
tests.integration.snowflake.test_snowflake ‑ test_snowflake_basic
tests.integration.snowflake.test_snowflake ‑ test_snowflake_private_link
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_list_columns_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_list_primary_keys_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_missing_role_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_missing_snowflake_lineage_permission_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_missing_snowflake_operations_permission_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_missing_warehouse_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_no_databases_with_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_no_tables_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures ‑ test_snowflake_unexpected_snowflake_view_lineage_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_list_columns_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_list_primary_keys_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_missing_role_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_missing_snowflake_lineage_permission_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_missing_snowflake_operations_permission_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_missing_warehouse_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_no_databases_with_access_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_no_tables_causes_pipeline_failure
tests.integration.snowflake.test_snowflake_failures_legacy_lineage ‑ test_snowflake_unexpected_snowflake_view_lineage_error_causes_pipeline_warning
tests.integration.snowflake.test_snowflake_legacy_lineage ‑ test_snowflake_basic
tests.integration.snowflake.test_snowflake_legacy_lineage ‑ test_snowflake_private_link
tests.integration.sql_server.test_sql_server ‑ test_mssql_ingest[mssql_no_db_to_file.yml]
tests.integration.sql_server.test_sql_server ‑ test_mssql_ingest[mssql_no_db_with_filter.yml]
tests.integration.sql_server.test_sql_server ‑ test_mssql_ingest[mssql_to_file.yml]
tests.integration.starburst-trino-usage.test_starburst_trino_usage ‑ test_trino_usage_config
tests.integration.starburst-trino-usage.test_starburst_trino_usage ‑ test_trino_usage_source
tests.integration.superset.test_superset ‑ test_superset_ingest
tests.integration.superset.test_superset ‑ test_superset_stateful_ingest
tests.integration.tableau.test_tableau_ingest ‑ test_extract_all_project
tests.integration.tableau.test_tableau_ingest ‑ test_lineage_overrides
tests.integration.tableau.test_tableau_ingest ‑ test_project_hierarchy
tests.integration.tableau.test_tableau_ingest ‑ test_project_path_pattern
tests.integration.tableau.test_tableau_ingest ‑ test_project_pattern
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_ingest
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_ingest_with_platform_instance
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_no_verify
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_signout_timeout
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_stateful
tests.integration.tableau.test_tableau_ingest ‑ test_tableau_unsupported_csql
tests.integration.tableau.test_tableau_ingest ‑ test_value_error_projects_and_project_pattern
tests.integration.trino.test_trino ‑ test_trino_hive_ingest
tests.integration.trino.test_trino ‑ test_trino_ingest
tests.integration.trino.test_trino ‑ test_trino_instance_ingest
tests.integration.unity.test_unity_catalog_ingest ‑ test_ingestion
tests.integration.vertica.test_vertica ‑ test_vertica_ingest_with_db
tests.unit.api.entities.dataproducts.test_dataproduct ‑ test_dataproduct_from_datahub
tests.unit.api.entities.dataproducts.test_dataproduct ‑ test_dataproduct_from_yaml[update]
tests.unit.api.entities.dataproducts.test_dataproduct ‑ test_dataproduct_from_yaml[upsert]
tests.unit.api.entities.dataproducts.test_dataproduct ‑ test_dataproduct_patch_yaml[asset_add]
tests.unit.api.entities.dataproducts.test_dataproduct ‑ test_dataproduct_patch_yaml[asset_remove]
tests.unit.api.test_report ‑ test_report_to_string_sampled
tests.unit.api.test_report ‑ test_report_to_string_unsampled
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_forced_not_a_version_tag
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_forced_stable
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_get_older_version
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_version_config
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_version_config_default
tests.unit.cli.test_quickstart_version_mapping ‑ test_quickstart_version_config_stable
tests.unit.config.test_config_enum ‑ test_config_enum
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_complex_variable_expansion.yml-env3-MissingClosingBrace]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_extension.whatevenisthis-env2-ConfigurationError]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/bad_variable_expansion.yml-env0-ParameterNullOrNotSet]
tests.unit.config.test_config_loader ‑ test_load_error[tests/unit/config/this_file_does_not_exist.yml-env1-ConfigurationError]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/basic.toml-golden_config1-env1-None]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/basic.yml-golden_config0-env0-referenced_env_vars0]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/complex_variable_expansion.yml-golden_config3-env3-referenced_env_vars3]
tests.unit.config.test_config_loader ‑ test_load_success[tests/unit/config/simple_variable_expansion.yml-golden_config2-env2-referenced_env_vars2]
tests.unit.config.test_config_model ‑ test_config_redaction
tests.unit.config.test_config_model ‑ test_default_object_copy
tests.unit.config.test_config_model ‑ test_extras_allowed
tests.unit.config.test_config_model ‑ test_extras_not_allowed
tests.unit.config.test_config_model ‑ test_shared_defaults
tests.unit.config.test_datetime_parser ‑ test_user_time_parser
tests.unit.data_lake.test_schema_inference ‑ test_infer_schema_avro
tests.unit.data_lake.test_schema_inference ‑ test_infer_schema_csv
tests.unit.data_lake.test_schema_inference ‑ test_infer_schema_json
tests.unit.data_lake.test_schema_inference ‑ test_infer_schema_parquet
tests.unit.data_lake.test_schema_inference ‑ test_infer_schema_tsv
tests.unit.graph.test_client ‑ test_get_aspect
tests.unit.graph.test_client ‑ test_graphql_entity_types
tests.unit.reporting.test_datahub_ingestion_reporter ‑ test_default_config
tests.unit.reporting.test_datahub_ingestion_reporter ‑ test_unique_key_gen[all_things]
tests.unit.reporting.test_datahub_ingestion_reporter ‑ test_unique_key_gen[minimal]
tests.unit.reporting.test_datahub_ingestion_reporter ‑ test_unique_key_gen[with_pipeline_name]
tests.unit.s3.test_s3_source ‑ test_partition_comparator_numeric_folder_name
tests.unit.s3.test_s3_source ‑ test_partition_comparator_numeric_folder_name2
tests.unit.s3.test_s3_source ‑ test_partition_comparator_string_folder
tests.unit.s3.test_s3_source ‑ test_partition_comparator_string_same_folder
tests.unit.s3.test_s3_source ‑ test_partition_comparator_with_equal_sign_in_name
tests.unit.s3.test_s3_source ‑ test_partition_comparator_with_numeric_partition
tests.unit.s3.test_s3_source ‑ test_partition_comparator_with_padded_numeric_partition
tests.unit.s3.test_s3_source ‑ test_partition_comparator_with_string_partition
tests.unit.sagemaker.test_sagemaker_source ‑ test_sagemaker_ingest
tests.unit.schema.test_json_schema_util ‑ test_array_handling
tests.unit.schema.test_json_schema_util ‑ test_datahub_json_schemas_parses_okay
tests.unit.schema.test_json_schema_util ‑ test_ignore_exceptions
tests.unit.schema.test_json_schema_util ‑ test_json_sample_payment_schema_to_schema_fields_with_nesting
tests.unit.schema.test_json_schema_util ‑ test_json_schema_to_events_with_nullable_fields[optional_field_via_union_type]
tests.unit.schema.test_json_schema_util ‑ test_json_schema_to_mce_fields_sample_events_with_different_field_types
tests.unit.schema.test_json_schema_util ‑ test_json_schema_to_mce_fields_toplevel_isnt_a_record
tests.unit.schema.test_json_schema_util ‑ test_json_schema_to_record_with_two_fields
tests.unit.schema.test_json_schema_util ‑ test_json_schema_to_schema_fields_with_nesting_across_records
tests.unit.schema.test_json_schema_util ‑ test_json_schema_with_recursion
tests.unit.schema.test_json_schema_util ‑ test_key_schema_handling
tests.unit.schema.test_json_schema_util ‑ test_map_of_union_of_int_and_record_of_union
tests.unit.schema.test_json_schema_util ‑ test_needs_disambiguation_nested_union_of_records_with_same_field_name
tests.unit.schema.test_json_schema_util ‑ test_nested_arrays
tests.unit.schema.test_json_schema_util ‑ test_recursive_json
tests.unit.schema.test_json_schema_util ‑ test_required_field
tests.unit.schema.test_json_schema_util ‑ test_simple_array
tests.unit.schema.test_json_schema_util ‑ test_simple_nested_record_with_a_string_field_for_key_schema
tests.unit.schema.test_json_schema_util ‑ test_simple_object
tests.unit.schema.test_json_schema_util ‑ test_simple_record_with_primitive_types
tests.unit.schema.test_json_schema_util ‑ test_union_with_nested_record_of_union
tests.unit.serde.test_serde ‑ test_check_mce_schema_failure[tests/unit/serde/test_serde_missing_field.json]
tests.unit.serde.test_serde ‑ test_check_metadata_rewrite
tests.unit.serde.test_serde ‑ test_check_metadata_schema[examples/mce_files/bootstrap_mce.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[examples/mce_files/mce_list.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[examples/mce_files/single_mce.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[tests/unit/serde/test_serde_backwards_compat.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[tests/unit/serde/test_serde_large.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[tests/unit/serde/test_serde_profile.json]
tests.unit.serde.test_serde ‑ test_check_metadata_schema[tests/unit/serde/test_serde_usage.json]
tests.unit.serde.test_serde ‑ test_field_discriminator
tests.unit.serde.test_serde ‑ test_json_transforms[model0-ref_server_obj0]
tests.unit.serde.test_serde ‑ test_missing_optional_in_union
tests.unit.serde.test_serde ‑ test_missing_optional_simple
tests.unit.serde.test_serde ‑ test_null_hiding
tests.unit.serde.test_serde ‑ test_read_empty_dict
tests.unit.serde.test_serde ‑ test_reserved_keywords
tests.unit.serde.test_serde ‑ test_serde_to_avro[tests/unit/serde/test_serde_chart_snapshot.json]
tests.unit.serde.test_serde ‑ test_serde_to_avro[tests/unit/serde/test_serde_extra_field.json]
tests.unit.serde.test_serde ‑ test_serde_to_avro[tests/unit/serde/test_serde_large.json]
tests.unit.serde.test_serde ‑ test_serde_to_json[tests/unit/serde/test_serde_chart_snapshot.json]
tests.unit.serde.test_serde ‑ test_serde_to_json[tests/unit/serde/test_serde_large.json]
tests.unit.serde.test_serde ‑ test_serde_to_json[tests/unit/serde/test_serde_profile.json]
tests.unit.serde.test_serde ‑ test_serde_to_json[tests/unit/serde/test_serde_usage.json]
tests.unit.serde.test_serde ‑ test_type_error
tests.unit.serde.test_serde ‑ test_unions_with_aliases_assumptions
tests.unit.serde.test_serde ‑ test_write_optional_empty_dict
tests.unit.serde.test_urn_iterator ‑ test_dataset_urn_lowercase_transformer
tests.unit.serde.test_urn_iterator ‑ test_list_urns_upstream
tests.unit.serde.test_urn_iterator ‑ test_upstream_lineage_urn_iterator
tests.unit.stateful_ingestion.provider.test_datahub_ingestion_checkpointing_provider.TestDatahubIngestionCheckpointProvider ‑ test_provider
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_base85_upgrade_pickle_to_json
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_checkpoint_serde[BaseSQLAlchemyCheckpointState]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_checkpoint_serde[BaseUsageCheckpointState]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_serde_idempotence[BaseSQLAlchemyCheckpointState]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_serde_idempotence[BaseUsageCheckpointState]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_state_forward_compatibility[base85-bz2-json]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_state_forward_compatibility[utf-8]
tests.unit.stateful_ingestion.state.test_checkpoint ‑ test_supported_encodings
tests.unit.stateful_ingestion.state.test_sql_common_state ‑ test_deduplication_and_order_preservation
tests.unit.stateful_ingestion.state.test_sql_common_state ‑ test_sql_common_state
tests.unit.stateful_ingestion.state.test_sql_common_state ‑ test_state_backward_compat
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[change_100_percent_delta_empty_new]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[change_100_percent_delta_non_empty_new]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[change_25_percent_delta]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[change_50_percent_delta]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[change_75_percent_delta]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[no_change_empty_old_and_new]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[no_change_empty_old_and_non_empty_new]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[no_change_non_empty_old_new_equals_old]
tests.unit.stateful_ingestion.state.test_stale_entity_removal_handler ‑ test_change_percent[no_change_non_empty_old_new_superset_old]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[checkpointing_bad_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[checkpointing_default]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[checkpointing_valid_full_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[checkpointing_valid_simple_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[stateful_ingestion_bad_config]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[stateful_ingestion_default_disabled]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[stateful_ingestion_default_enabled]
tests.unit.stateful_ingestion.test_configs ‑ test_state_provider_configs[stateful_ingestion_full_custom]
tests.unit.stateful_ingestion.test_kafka_state ‑ test_kafka_common_state
tests.unit.stateful_ingestion.test_kafka_state ‑ test_kafka_state_migration
tests.unit.test_airflow ‑ test_airflow_provider_info
tests.unit.test_airflow ‑ test_dags_load_with_no_errors
tests.unit.test_airflow ‑ test_datahub_kafka_hook
tests.unit.test_airflow ‑ test_datahub_lineage_operator
tests.unit.test_airflow ‑ test_datahub_rest_hook
tests.unit.test_airflow ‑ test_datahub_rest_hook_with_timeout
tests.unit.test_airflow ‑ test_entities
tests.unit.test_airflow ‑ test_hook_airflow_ui[DatahubKafkaHook]
tests.unit.test_airflow ‑ test_hook_airflow_ui[DatahubRestHook]
tests.unit.test_airflow ‑ test_lineage_backend[airflow-lineage-capture-executions]
tests.unit.test_airflow ‑ test_lineage_backend[airflow-lineage-no-executions]
tests.unit.test_allow_deny ‑ test_allow_all
tests.unit.test_allow_deny ‑ test_case_sensitivity
tests.unit.test_allow_deny ‑ test_default_deny
tests.unit.test_allow_deny ‑ test_deny_all
tests.unit.test_allow_deny ‑ test_fully_speced
tests.unit.test_allow_deny ‑ test_is_allowed
tests.unit.test_allow_deny ‑ test_prefix_match
tests.unit.test_allow_deny ‑ test_single_table
tests.unit.test_apis ‑ test_sinks_not_abstract
tests.unit.test_apis ‑ test_sources_not_abstract
tests.unit.test_athena_source ‑ test_athena_config_query_location_old_plus_new_value_not_allowed
tests.unit.test_athena_source ‑ test_athena_config_staging_dir_is_set_as_query_result
tests.unit.test_athena_source ‑ test_athena_get_table_properties
tests.unit.test_athena_source ‑ test_athena_uri
tests.unit.test_bigquery_lineage ‑ test_create_statement_with_multiple_table
tests.unit.test_bigquery_lineage ‑ test_lineage_with_timestamps
tests.unit.test_bigquery_lineage ‑ test_one_part_table
tests.unit.test_bigquery_lineage ‑ test_parse_view_lineage
tests.unit.test_bigquery_lineage ‑ test_parse_view_lineage_with_two_part_table_name
tests.unit.test_bigquery_profiler ‑ test_generate_day_partitioned_partition_profiler_query
tests.unit.test_bigquery_profiler ‑ test_generate_day_partitioned_partition_profiler_query_with_set_partition_time
tests.unit.test_bigquery_profiler ‑ test_generate_hour_partitioned_partition_profiler_query
tests.unit.test_bigquery_profiler ‑ test_generate_ingestion_partitioned_partition_profiler_query
tests.unit.test_bigquery_profiler ‑ test_generate_sharded_table_profiler_query
tests.unit.test_bigquery_profiler ‑ test_not_generate_partition_profiler_query_if_not_partitioned_sharded_table
tests.unit.test_bigquery_source ‑ test_bigquery_uri
tests.unit.test_bigquery_source ‑ test_bigquery_uri_on_behalf
tests.unit.test_bigquery_source ‑ test_bigquery_uri_with_credential
tests.unit.test_bigquery_source ‑ test_gen_view_dataset_workunits
tests.unit.test_bigquery_source ‑ test_get_projects_by_list
tests.unit.test_bigquery_source ‑ test_get_projects_filter_by_pattern
tests.unit.test_bigquery_source ‑ test_get_projects_list_empty
tests.unit.test_bigquery_source ‑ test_get_projects_list_failure
tests.unit.test_bigquery_source ‑ test_get_projects_list_fully_filtered
tests.unit.test_bigquery_source ‑ test_get_projects_with_project_ids
tests.unit.test_bigquery_source ‑ test_get_projects_with_project_ids_overrides_project_id_pattern
tests.unit.test_bigquery_source ‑ test_get_projects_with_single_project_id
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[2023-None-2023]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[20231215-None-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[project.dataset.2023-project.dataset.2023-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[project.dataset.20231215-project.dataset.20231215-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[project.dataset.table-project.dataset.table-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[project.dataset.table_2023-project.dataset.table-2023]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[project.dataset.table_20231215-project.dataset.table-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table-table-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table20231215-table20231215-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table_1624046611-table-1624046611]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table_1624046611000-table_1624046611000-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table_1624046611000_name-table_1624046611000_name-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table_2023-table-2023]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_custom_shard_pattern[table_20231215-table-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[20231215-None-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[project.dataset.20231215-project.dataset.20231215-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[project.dataset.table-project.dataset.table-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[project.dataset.table_2023-project.dataset.table_2023-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[project.dataset.table_20231215-project.dataset.table-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[table-table-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[table20231215-table20231215-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[table_1624046611000-table_1624046611000-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[table_1624046611000_name-table_1624046611000_name-None]
tests.unit.test_bigquery_source ‑ test_get_table_and_shard_default[table_20231215-table-20231215]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.20230112-project.dataset.dataset]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table20231215-project.dataset.table20231215]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table@1624046611000-1612046611000-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_table_name[[email protected]]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_*-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_1624046611000-project.dataset.table_1624046611000]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_1624046611000_name-project.dataset.table_1624046611000_name]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_2023*-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_202301*-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_table_name[project.dataset.table_20231215-project.dataset.table]
tests.unit.test_bigquery_source ‑ test_get_views_for_dataset
tests.unit.test_bigquery_source ‑ test_simple_upstream_table_generation
tests.unit.test_bigquery_source ‑ test_table_processing_logic
tests.unit.test_bigquery_source ‑ test_table_processing_logic_date_named_tables
tests.unit.test_bigquery_source ‑ test_upstream_table_generation_with_temporary_table_with_multiple_temp_upstream
tests.unit.test_bigquery_source ‑ test_upstream_table_generation_with_temporary_table_with_temp_upstream
tests.unit.test_bigquery_source ‑ test_upstream_table_generation_with_temporary_table_without_temp_upstream
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_dataset
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_table
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_table_and_dataset
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_table_and_dataset_joins
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_table_and_dataset_joins_and_subquery
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_camel_case_table_and_dataset_subquery
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_create_or_replace_view_name_with_hyphens_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_cte_alias_as_keyword_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_from_as_column_name_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_hash_as_comment_sign_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_keyword_admin_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_keyword_data_is_accepted
tests.unit.test_bigquery_sql_lineage ‑ test_bigquery_sql_lineage_source_table_name_with_hyphens_is_accepted
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_comment_sign_switched_correctly
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_comments_are_removed
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_field_name_is_not_escaped_after_keyword_from_in_datetime_functions
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_first_cte_name_is_escaped
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_formats_input_sql
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_keyword_from_is_escaped_if_used_as_fieldname
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_object_name_is_escaped_after_keyword_from
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_subquery
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_table_name_is_escaped_at_create_statement
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_view_name_is_escaped_at_create_statement
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_with_parenthesis_in_from
tests.unit.test_bigquery_sql_parser ‑ test_bigquery_sql_parser_with_semicolon_in_from
tests.unit.test_bigquery_sql_parser ‑ test_remove_comma_before_from
tests.unit.test_bigquery_usage ‑ test_get_tables_from_query
tests.unit.test_bigquery_usage ‑ test_operational_stats
tests.unit.test_bigquery_usage ‑ test_usage_counts_multiple_buckets_and_resources_no_view_usage
tests.unit.test_bigquery_usage ‑ test_usage_counts_multiple_buckets_and_resources_view_usage
tests.unit.test_bigquery_usage ‑ test_usage_counts_no_columns
tests.unit.test_bigquery_usage ‑ test_usage_counts_no_query_event
tests.unit.test_bigquery_usage ‑ test_usage_counts_single_bucket_resource_project
tests.unit.test_bigqueryv2_usage_source ‑ test_bigquery_table_sanitasitation
tests.unit.test_bigqueryv2_usage_source ‑ test_bigqueryv2_filters
tests.unit.test_bigqueryv2_usage_source ‑ test_bigqueryv2_uri_with_credential
tests.unit.test_bq_get_partition_range ‑ test_get_partition_range_from_partition_id
tests.unit.test_capability_report ‑ test_basic_capability_report
tests.unit.test_cdh_hive.CDHHiveSourceTest ‑ test_cdh_hive_source_configuration
tests.unit.test_cdh_hive.CDHHiveSourceTest ‑ test_hive_source_workunits
tests.unit.test_check ‑ test_check_local_docker
tests.unit.test_check ‑ test_cli_help
tests.unit.test_check ‑ test_cli_version
tests.unit.test_check_upgrade ‑ test_is_client_server_compatible
tests.unit.test_classification ‑ test_default_classification_config
tests.unit.test_classification ‑ test_default_datahub_classifier_config
tests.unit.test_classification ‑ test_selective_datahub_classifier_config_override
tests.unit.test_cli_logging ‑ test_cli_logging
tests.unit.test_cli_logging ‑ test_extra_args_exception_suppressed
tests.unit.test_cli_utils ‑ test_correct_url_when_gms_host_and_port_set
tests.unit.test_cli_utils ‑ test_correct_url_when_gms_host_in_old_format
tests.unit.test_cli_utils ‑ test_correct_url_when_gms_host_port_url_protocol_set
tests.unit.test_cli_utils ‑ test_correct_url_when_gms_host_port_url_set
tests.unit.test_cli_utils ‑ test_correct_url_when_url_set
tests.unit.test_cli_utils ‑ test_first_non_null
tests.unit.test_clickhouse_source ‑ test_clickhouse_uri_default_password
tests.unit.test_clickhouse_source ‑ test_clickhouse_uri_https
tests.unit.test_clickhouse_source ‑ test_clickhouse_uri_native
tests.unit.test_clickhouse_source ‑ test_clickhouse_uri_native_secure
tests.unit.test_codegen ‑ test_cannot_instantiate_codegen_aspect
tests.unit.test_codegen ‑ test_class_filter
tests.unit.test_codegen ‑ test_codegen_aspect_name
tests.unit.test_codegen ‑ test_codegen_aspects
tests.unit.test_codegen ‑ test_entity_registry_completeness
tests.unit.test_codegen ‑ test_key_aspect_info
tests.unit.test_codegen ‑ test_urn_annotation
tests.unit.test_config_clean ‑ test_remove_protocol_http
tests.unit.test_config_clean ‑ test_remove_protocol_http_accidental_multiple
tests.unit.test_config_clean ‑ test_remove_protocol_https
tests.unit.test_config_clean ‑ test_remove_suffix
tests.unit.test_config_clean ‑ test_url_with_multiple_slashes
tests.unit.test_config_clean ‑ test_url_with_suffix
tests.unit.test_config_clean ‑ test_url_without_slash_suffix
tests.unit.test_confluent_schema_registry.ConfluentSchemaRegistryTest ‑ test_get_schema_str_replace_confluent_ref_avro
tests.unit.test_corp_group_urn.TestCorpGroupUrn ‑ test_invalid_urn
tests.unit.test_corp_group_urn.TestCorpGroupUrn ‑ test_parse_urn
tests.unit.test_corpuser_urn.TestCorpuserUrn ‑ test_invalid_urn
tests.unit.test_corpuser_urn.TestCorpuserUrn ‑ test_parse_urn
tests.unit.test_csv_enricher_source ‑ test_get_resource_description_no_description
tests.unit.test_csv_enricher_source ‑ test_get_resource_description_work_unit_produced
tests.unit.test_csv_enricher_source ‑ test_get_resource_domain_no_domain
tests.unit.test_csv_enricher_source ‑ test_get_resource_domain_work_unit_produced
tests.unit.test_csv_enricher_source ‑ test_get_resource_glossary_terms_no_new_glossary_terms
tests.unit.test_csv_enricher_source ‑ test_get_resource_glossary_terms_work_unit_no_terms
tests.unit.test_csv_enricher_source ‑ test_get_resource_glossary_terms_work_unit_produced
tests.unit.test_csv_enricher_source ‑ test_get_resource_owners_no_new_owners
tests.unit.test_csv_enricher_source ‑ test_get_resource_owners_work_unit_no_terms
tests.unit.test_csv_enricher_source ‑ test_get_resource_owners_work_unit_produced
tests.unit.test_csv_enricher_source ‑ test_get_resource_tags_no_new_tags
tests.unit.test_csv_enricher_source ‑ test_get_resource_tags_work_unit_no_tags
tests.unit.test_csv_enricher_source ‑ test_get_resource_tags_work_unit_produced
tests.unit.test_data_flow_urn.TestDataFlowUrn ‑ test_invalid_urn
tests.unit.test_data_flow_urn.TestDataFlowUrn ‑ test_parse_urn
tests.unit.test_data_job_urn.TestDataJobUrn ‑ test_invalid_urn
tests.unit.test_data_job_urn.TestDataJobUrn ‑ test_parse_urn
tests.unit.test_data_process_instance_urn.TestDomainUrn ‑ test_invalid_urn
tests.unit.test_data_process_instance_urn.TestDomainUrn ‑ test_parse_urn
tests.unit.test_dataset_urn.TestDatasetUrn ‑ test_invalid_urn
tests.unit.test_dataset_urn.TestDatasetUrn ‑ test_parse_urn
tests.unit.test_dbt_source ‑ test_dbt_entity_emission_configuration
tests.unit.test_dbt_source ‑ test_dbt_entity_emission_configuration_helpers
tests.unit.test_dbt_source ‑ test_dbt_source_patching_no_conflict
tests.unit.test_dbt_source ‑ test_dbt_source_patching_no_new
tests.unit.test_dbt_source ‑ test_dbt_source_patching_tags
tests.unit.test_dbt_source ‑ test_dbt_source_patching_terms
tests.unit.test_dbt_source ‑ test_dbt_source_patching_with_conflict
tests.unit.test_dbt_source ‑ test_dbt_source_patching_with_conflict_null_source_type_in_existing_owner
tests.unit.test_dbt_source ‑ test_default_convert_column_urns_to_lowercase
tests.unit.test_domain_urn.TestDomainUrn ‑ test_invalid_urn
tests.unit.test_domain_urn.TestDomainUrn ‑ test_parse_urn
tests.unit.test_druid_source ‑ test_druid_uri
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[.ds-datahub_usage_event-000001]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[chartindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[corpgroupindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[corpuserindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dashboardindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dataflowindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[datahubpolicyindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[datahubretentionindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[datajobindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dataplatformindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dataprocessindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dataset_datasetprofileaspect_v1]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[dataset_datasetusagestatisticsaspect_v1]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[datasetindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[glossarynodeindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[glossarytermindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[ilm-history-2-000001]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlfeatureindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlfeaturetableindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlmodeldeploymentindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlmodelgroupindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlmodelindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[mlprimarykeyindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[schemafieldindex_v2]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[system_metadata_service_v1]
tests.unit.test_elasticsearch_source ‑ test_elastic_search_schema_conversion[tagindex_v2]
tests.unit.test_elasticsearch_source ‑ test_host_port_parsing
tests.unit.test_elasticsearch_source ‑ test_no_properties_in_mappings_schema
tests.unit.test_file_lineage_source ‑ test_basic_lineage_entity_root_node_urn
tests.unit.test_file_lineage_source ‑ test_basic_lineage_upstream_urns
tests.unit.test_file_lineage_source ‑ test_unsupported_entity_env
tests.unit.test_file_lineage_source ‑ test_unsupported_entity_type
tests.unit.test_file_lineage_source ‑ test_unsupported_upstream_entity_type
tests.unit.test_gcs_source ‑ test_data_lake_incorrect_config_raises_error
tests.unit.test_gcs_source ‑ test_gcs_source_setup
tests.unit.test_generic_aspect_transformer.TestDummyGenericAspectTransformer ‑ test_add_generic_aspect_when_mce_received
tests.unit.test_generic_aspect_transformer.TestDummyGenericAspectTransformer ‑ test_add_generic_aspect_when_mcpc_received
tests.unit.test_generic_aspect_transformer.TestDummyGenericAspectTransformer ‑ test_add_generic_aspect_when_mcpw_received
tests.unit.test_generic_aspect_transformer.TestDummyGenericAspectTransformer ‑ test_modify_generic_aspect_when_mcpc_received
tests.unit.test_generic_aspect_transformer.TestDummyRemoveGenericAspectTransformer ‑ test_remove_generic_aspect_when_mcpc_received
tests.unit.test_glue_source ‑ test_column_type[array]
tests.unit.test_glue_source ‑ test_column_type[char]
tests.unit.test_glue_source ‑ test_column_type[map]
tests.unit.test_glue_source ‑ test_column_type[struct]
tests.unit.test_glue_source ‑ test_config_without_platform
tests.unit.test_glue_source ‑ test_glue_ingest[None-glue_mces.json-glue_mces_golden.json]
tests.unit.test_glue_source ‑ test_glue_ingest[some_instance_name-glue_mces_platform_instance.json-glue_mces_platform_instance_golden.json]
tests.unit.test_glue_source ‑ test_glue_stateful
tests.unit.test_glue_source ‑ test_ignore_resource_links[False-all_databases_and_tables_result1]
tests.unit.test_glue_source ‑ test_ignore_resource_links[True-all_databases_and_tables_result0]
tests.unit.test_glue_source ‑ test_platform_config
tests.unit.test_glue_source ‑ test_platform_must_be_valid
tests.unit.test_great_expectations_action ‑ test_DataHubValidationAction_basic
tests.unit.test_great_expectations_action ‑ test_DataHubValidationAction_graceful_failure
tests.unit.test_great_expectations_action ‑ test_DataHubValidationAction_not_supported
tests.unit.test_hana_source ‑ test_hana_uri_native
tests.unit.test_hana_source ‑ test_hana_uri_native_db
tests.unit.test_hana_source ‑ test_platform_correctly_set_hana
tests.unit.test_hive_source ‑ test_hive_configuration_get_avro_schema_from_native_data_type
tests.unit.test_hive_source ‑ test_hive_configuration_get_identifier_with_database
tests.unit.test_iceberg ‑ test_adls_config_no_credential
tests.unit.test_iceberg ‑ test_adls_config_with_client_secret_credential
tests.unit.test_iceberg ‑ test_adls_config_with_key_credential
tests.unit.test_iceberg ‑ test_adls_config_with_sas_credential
tests.unit.test_iceberg ‑ test_avro_decimal_bytes_nullable
tests.unit.test_iceberg ‑ test_config_for_tests
tests.unit.test_iceberg ‑ test_config_multiple_filesystems
tests.unit.test_iceberg ‑ test_config_no_filesystem
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type0-bytes]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type1-boolean]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type10-timestamp-micros]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type11-timestamp-micros]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type12-time-micros]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type13-uuid]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type2-date]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type3-decimal]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type4-double]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type5-fixed]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type6-float]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type7-int]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type8-long]
tests.unit.test_iceberg ‑ test_iceberg_list_to_schema_field[iceberg_type9-string]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type0-BytesTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type1-BooleanTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type10-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type11-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type12-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type13-StringTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type2-DateTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type3-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type4-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type5-FixedTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type6-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type7-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type8-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_map_to_schema_field[iceberg_type9-StringTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type0-BytesTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type1-BooleanTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type10-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type11-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type12-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type13-StringTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type2-DateTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type3-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type4-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type5-FixedTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type6-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type7-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type8-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_primitive_type_to_schema_field[iceberg_type9-StringTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type0-BytesTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type1-BooleanTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type10-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type11-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type12-TimeTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type13-StringTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type2-DateTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type3-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type4-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type5-FixedTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type6-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type7-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type8-NumberTypeClass]
tests.unit.test_iceberg ‑ test_iceberg_struct_to_schema_field[iceberg_type9-StringTypeClass]
tests.unit.test_kafka_emitter.KafkaEmitterTest ‑ test_kafka_emitter_config
tests.unit.test_kafka_emitter.KafkaEmitterTest ‑ test_kafka_emitter_config_old_and_new
tests.unit.test_kafka_emitter.KafkaEmitterTest ‑ test_kafka_emitter_config_topic_upgrade
tests.unit.test_kafka_sink.KafkaSinkTest ‑ test_kafka_callback_class
tests.unit.test_kafka_sink.KafkaSinkTest ‑ test_kafka_sink_close
tests.unit.test_kafka_sink.KafkaSinkTest ‑ test_kafka_sink_config
tests.unit.test_kafka_sink.KafkaSinkTest ‑ test_kafka_sink_mcp
tests.unit.test_kafka_sink.KafkaSinkTest ‑ test_kafka_sink_write
tests.unit.test_kafka_source ‑ test_close
tests.unit.test_kafka_source ‑ test_kafka_ignore_warnings_on_schema_type[ignore_warnings_on_schema_type-FALSE]
tests.unit.test_kafka_source ‑ test_kafka_ignore_warnings_on_schema_type[ignore_warnings_on_schema_type-TRUE]
tests.unit.test_kafka_source ‑ test_kafka_source_configuration
tests.unit.test_kafka_source ‑ test_kafka_source_succeeds_with_admin_client_init_error
tests.unit.test_kafka_source ‑ test_kafka_source_succeeds_with_describe_configs_error
tests.unit.test_kafka_source ‑ test_kafka_source_workunits_no_platform_instance
tests.unit.test_kafka_source ‑ test_kafka_source_workunits_schema_registry_subject_name_strategies
tests.unit.test_kafka_source ‑ test_kafka_source_workunits_topic_pattern
tests.unit.test_kafka_source ‑ test_kafka_source_workunits_wildcard_topic
tests.unit.test_kafka_source ‑ test_kafka_source_workunits_with_platform_instance
tests.unit.test_key_value_pattern ‑ test_basic_pattern
tests.unit.test_key_value_pattern ‑ test_empty_pattern
tests.unit.test_key_value_pattern ‑ test_fallthrough_pattern
tests.unit.test_key_value_pattern ‑ test_fullmatch_mix_pattern
tests.unit.test_key_value_pattern ‑ test_fullmatch_pattern
tests.unit.test_key_value_pattern ‑ test_no_fallthrough_pattern
tests.unit.test_key_value_pattern ‑ test_regex_pattern
tests.unit.test_kudu.KuduSourceTest ‑ test_kudu_source_configuration
tests.unit.test_kudu.KuduSourceTest ‑ test_kudu_source_workunits
tests.unit.test_ldap_source ‑ test_parse_groups[input0-expected0]
tests.unit.test_ldap_source ‑ test_parse_groups[input1-expected1]
tests.unit.test_ldap_source ‑ test_parse_ldap_dn[cn=comma group (one\, two\, three),ou=Groups,dc=internal,dc=machines-comma group (one, two, three)]
tests.unit.test_ldap_source ‑ test_parse_ldap_dn[cn=group_name,ou=Groups,dc=internal,dc=machines-group_name]
tests.unit.test_ldap_source ‑ test_parse_ldap_dn[uid=firstname.surname,ou=People,dc=internal,dc=machines-firstname.surname]
tests.unit.test_ldap_source ‑ test_parse_users[input0-expected0]
tests.unit.test_ldap_source ‑ test_parse_users[input1-expected1]
tests.unit.test_ldap_state ‑ test_add_checkpoint_urn
tests.unit.test_ldap_state ‑ test_get_percent_entities_changed
tests.unit.test_ldap_state ‑ test_get_urns_not_in
tests.unit.test_mapping ‑ test_operation_processor_advanced_matching_owners
tests.unit.test_mapping ‑ test_operation_processor_advanced_matching_tags
tests.unit.test_mapping ‑ test_operation_processor_matching
tests.unit.test_mapping ‑ test_operation_processor_no_email_strip_source_type_not_null
tests.unit.test_mapping ‑ test_operation_processor_not_matching
tests.unit.test_mapping ‑ test_operation_processor_ownership_category
tests.unit.test_mariadb_source ‑ test_platform_correctly_set_mariadb
tests.unit.test_mariadb_source ‑ test_platform_correctly_set_mysql
tests.unit.test_mce_builder ‑ test_can_add_aspect

Check notice on line 0 in .github

See this annotation in the file changed.

@github-actions github-actions / Unit Test Results (metadata ingestion)

1068 tests found (test 757 to 1068) 

There are 1068 tests, see "Raw output" for the list of tests 757 to 1068.
Raw output 
tests.unit.test_mce_builder ‑ test_create_dataset_urn_with_reserved_chars
tests.unit.test_mce_helpers ‑ test_basic_diff_only_owner_change
tests.unit.test_mce_helpers ‑ test_basic_diff_owner_change
tests.unit.test_mce_helpers ‑ test_basic_diff_same
tests.unit.test_mcp_builder ‑ test_guid_generator
tests.unit.test_mcp_builder ‑ test_guid_generator_with_empty_instance
tests.unit.test_mcp_builder ‑ test_guid_generator_with_env
tests.unit.test_mcp_builder ‑ test_guid_generator_with_instance
tests.unit.test_mcp_builder ‑ test_guid_generator_with_instance_and_env
tests.unit.test_mcp_builder ‑ test_guid_generators
tests.unit.test_mcp_wrapper ‑ test_mcpw_from_obj
tests.unit.test_mcp_wrapper ‑ test_mcpw_inference
tests.unit.test_nifi_source ‑ test_client_cert_auth_failed
tests.unit.test_nifi_source ‑ test_client_cert_auth_without_client_cert_file
tests.unit.test_nifi_source ‑ test_failure_to_create_nifi_flow
tests.unit.test_nifi_source ‑ test_kerberos_auth_failed_to_get_token
tests.unit.test_nifi_source ‑ test_nifi_s3_provenance_event
tests.unit.test_nifi_source ‑ test_single_user_auth_failed_to_get_token
tests.unit.test_nifi_source ‑ test_single_user_auth_without_password
tests.unit.test_nifi_source ‑ test_single_user_auth_without_username_and_password
tests.unit.test_notebook_urn.TestNotebookUrn ‑ test_invalid_urn
tests.unit.test_notebook_urn.TestNotebookUrn ‑ test_parse_urn
tests.unit.test_openapi.TestExplodeDict ‑ test_d1
tests.unit.test_openapi.TestGetEndpoints ‑ test_get_endpoints_openapi20
tests.unit.test_openapi.TestGetEndpoints ‑ test_get_endpoints_openapi30
tests.unit.test_openapi.TestGuessing ‑ test_mul_cids
tests.unit.test_openapi.TestGuessing ‑ test_mul_ids
tests.unit.test_openapi.TestGuessing ‑ test_name_id
tests.unit.test_openapi.TestGuessing ‑ test_name_id2
tests.unit.test_openapi.TestGuessing ‑ test_no_good_guesses
tests.unit.test_openapi.TestGuessing ‑ test_no_k_f
tests.unit.test_openapi.TestGuessing ‑ test_one_cid
tests.unit.test_openapi.TestGuessing ‑ test_one_id
tests.unit.test_openapi.TestGuessing ‑ test_only_id
tests.unit.test_oracle_source ‑ test_oracle_config
tests.unit.test_packages ‑ test_package_list_match_inits
tests.unit.test_packaging ‑ test_datahub_version
tests.unit.test_parsing_util ‑ test_get_missing_key
tests.unit.test_parsing_util ‑ test_get_missing_key_any
tests.unit.test_pipeline.TestPipeline ‑ test_configure
tests.unit.test_pipeline.TestPipeline ‑ test_configure_with_file_sink_does_not_init_graph
tests.unit.test_pipeline.TestPipeline ‑ test_configure_with_rest_sink_initializes_graph
tests.unit.test_pipeline.TestPipeline ‑ test_configure_with_rest_sink_with_additional_props_initializes_graph
tests.unit.test_pipeline.TestPipeline ‑ test_configure_without_sink
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ALWAYS-no-warnings-no-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ALWAYS-with-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ALWAYS-with-warnings]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS-no-warnings-no-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS-with-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS-with-warnings]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS_AND_NO_WARNINGS-no-warnings-no-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS_AND_NO_WARNINGS-with-errors]
tests.unit.test_pipeline.TestPipeline ‑ test_pipeline_process_commits[ON_NO_ERRORS_AND_NO_WARNINGS-with-warnings]
tests.unit.test_pipeline.TestPipeline ‑ test_run_including_fake_transformation
tests.unit.test_pipeline.TestPipeline ‑ test_run_including_registered_transformation
tests.unit.test_plugin_system ‑ test_list_all[False]
tests.unit.test_plugin_system ‑ test_list_all[True]
tests.unit.test_plugin_system ‑ test_registry
tests.unit.test_plugin_system ‑ test_registry_defaults[registry0-expected0]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry1-expected1]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry2-expected2]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry3-expected3]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry4-expected4]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry5-expected5]
tests.unit.test_plugin_system ‑ test_registry_defaults[registry6-expected6]
tests.unit.test_postgres_source ‑ test_current_sqlalchemy_database_in_identifier
tests.unit.test_postgres_source ‑ test_database_alias_takes_precendence
tests.unit.test_postgres_source ‑ test_database_in_identifier
tests.unit.test_postgres_source ‑ test_get_inspectors_multiple_databases
tests.unit.test_postgres_source ‑ test_initial_database
tests.unit.test_postgres_source ‑ tests_get_inspectors_with_database_provided
tests.unit.test_postgres_source ‑ tests_get_inspectors_with_sqlalchemy_uri_provided
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_map
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_nestd_repeated
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_nested
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_repeated
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_with_complex_schema
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_with_single_empty_message
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_with_single_message_single_field_key_schema
tests.unit.test_protobuf_util ‑ test_protobuf_schema_to_mce_fields_with_two_messages_enum
tests.unit.test_protobuf_util ‑ test_protobuf_schema_with_recursive_type
tests.unit.test_pulsar_source.TestPulsarSchema ‑ test_pulsar_source_parse_pulsar_schema
tests.unit.test_pulsar_source.TestPulsarSource ‑ test_pulsar_source_get_token_jwt
tests.unit.test_pulsar_source.TestPulsarSource ‑ test_pulsar_source_get_token_oauth
tests.unit.test_pulsar_source.TestPulsarSource ‑ test_pulsar_source_get_workunits_all_tenant
tests.unit.test_pulsar_source.TestPulsarSource ‑ test_pulsar_source_get_workunits_custom_tenant
tests.unit.test_pulsar_source.TestPulsarSource ‑ test_pulsar_source_get_workunits_patterns
tests.unit.test_pulsar_source.TestPulsarSourceConfig ‑ test_pulsar_source_config_invalid_web_service_url_host
tests.unit.test_pulsar_source.TestPulsarSourceConfig ‑ test_pulsar_source_config_invalid_web_service_url_scheme
tests.unit.test_pulsar_source.TestPulsarSourceConfig ‑ test_pulsar_source_config_valid_web_service_url
tests.unit.test_pulsar_source.TestPulsarTopic ‑ test_pulsar_source_parse_topic_string
tests.unit.test_pydantic_validators ‑ test_field_rename
tests.unit.test_redash_source ‑ test_get_chart_snapshot_parse_table_names_from_sql
tests.unit.test_redash_source ‑ test_get_dashboard_snapshot_after_v10
tests.unit.test_redash_source ‑ test_get_dashboard_snapshot_before_v10
tests.unit.test_redash_source ‑ test_get_full_qualified_name
tests.unit.test_redash_source ‑ test_get_known_viz_chart_snapshot
tests.unit.test_redash_source ‑ test_get_unknown_viz_chart_snapshot
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query_with_database
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query_with_four_part_table_should_throw_exception
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query_with_non_default_database
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query_with_only_table
tests.unit.test_redshift_lineage ‑ test_get_sources_from_query_with_only_table_name
tests.unit.test_report ‑ test_entity_filter_report
tests.unit.test_rest_emitter ‑ test_datahub_rest_emitter_construction
tests.unit.test_rest_emitter ‑ test_datahub_rest_emitter_extra_params
tests.unit.test_rest_emitter ‑ test_datahub_rest_emitter_retry_construction
tests.unit.test_rest_emitter ‑ test_datahub_rest_emitter_timeout_construction
tests.unit.test_rest_sink ‑ test_datahub_rest_emitter[record0-/entities?action=ingest-snapshot0]
tests.unit.test_rest_sink ‑ test_datahub_rest_emitter[record1-/entities?action=ingest-snapshot1]
tests.unit.test_rest_sink ‑ test_datahub_rest_emitter[record2-/entities?action=ingest-snapshot2]
tests.unit.test_rest_sink ‑ test_datahub_rest_emitter[record3-/usageStats?action=batchIngest-snapshot3]
tests.unit.test_rest_sink ‑ test_datahub_rest_emitter[record4-/aspects?action=ingestProposal-snapshot4]
tests.unit.test_schema_util ‑ test_avro_sample_payment_schema_to_mce_fields_with_nesting
tests.unit.test_schema_util ‑ test_avro_schema_namespacing
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_events_with_nullable_fields[optional_field_via_fixed]
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_events_with_nullable_fields[optional_field_via_primitive]
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_events_with_nullable_fields[optional_field_via_union_null_not_first]
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_events_with_nullable_fields[optional_field_via_union_type]
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_record_with_two_fields
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_sample_events_with_different_field_types
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_toplevel_isnt_a_record
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_with_default
tests.unit.test_schema_util ‑ test_avro_schema_to_mce_fields_with_nesting_across_records
tests.unit.test_schema_util ‑ test_avro_schema_with_recursion
tests.unit.test_schema_util ‑ test_ignore_exceptions
tests.unit.test_schema_util ‑ test_key_schema_handling
tests.unit.test_schema_util ‑ test_logical_types_bare
tests.unit.test_schema_util ‑ test_logical_types_fully_specified_in_type
tests.unit.test_schema_util ‑ test_map_of_union_of_int_and_record_of_union
tests.unit.test_schema_util ‑ test_mce_avro_parses_okay
tests.unit.test_schema_util ‑ test_needs_disambiguation_nested_union_of_records_with_same_field_name
tests.unit.test_schema_util ‑ test_nested_arrays
tests.unit.test_schema_util ‑ test_recursive_avro
tests.unit.test_schema_util ‑ test_simple_nested_record_with_a_string_field_for_key_schema
tests.unit.test_schema_util ‑ test_simple_record_with_primitive_types
tests.unit.test_schema_util ‑ test_union_with_nested_record_of_union
tests.unit.test_snowflake_source ‑ test_account_id_is_added_when_host_port_is_present
tests.unit.test_snowflake_source ‑ test_account_id_with_snowflake_host_suffix
tests.unit.test_snowflake_source ‑ test_aws_cloud_region_from_snowflake_region_id
tests.unit.test_snowflake_source ‑ test_azure_cloud_region_from_snowflake_region_id
tests.unit.test_snowflake_source ‑ test_google_cloud_region_from_snowflake_region_id
tests.unit.test_snowflake_source ‑ test_options_contain_connect_args
tests.unit.test_snowflake_source ‑ test_private_key_set_but_auth_not_changed
tests.unit.test_snowflake_source ‑ test_snowflake_config_with_column_lineage_no_table_lineage_throws_error
tests.unit.test_snowflake_source ‑ test_snowflake_config_with_connect_args_overrides_base_connect_args
tests.unit.test_snowflake_source ‑ test_snowflake_config_with_no_connect_args_returns_base_connect_args
tests.unit.test_snowflake_source ‑ test_snowflake_config_with_view_lineage_no_table_lineage_throws_error
tests.unit.test_snowflake_source ‑ test_snowflake_object_access_entry_missing_object_id
tests.unit.test_snowflake_source ‑ test_snowflake_query_create_deny_regex_sql
tests.unit.test_snowflake_source ‑ test_snowflake_source_throws_error_on_account_id_missing
tests.unit.test_snowflake_source ‑ test_snowflake_throws_error_on_client_id_missing_if_using_oauth
tests.unit.test_snowflake_source ‑ test_snowflake_throws_error_on_client_secret_missing_if_use_certificate_is_false
tests.unit.test_snowflake_source ‑ test_snowflake_throws_error_on_encoded_oauth_private_key_missing_if_use_certificate_is_true
tests.unit.test_snowflake_source ‑ test_snowflake_uri_default_authentication
tests.unit.test_snowflake_source ‑ test_snowflake_uri_external_browser_authentication
tests.unit.test_snowflake_source ‑ test_snowflake_uri_key_pair_authentication
tests.unit.test_snowflake_source ‑ test_test_connection_basic_success
tests.unit.test_snowflake_source ‑ test_test_connection_capability_all_success
tests.unit.test_snowflake_source ‑ test_test_connection_capability_schema_failure
tests.unit.test_snowflake_source ‑ test_test_connection_capability_schema_success
tests.unit.test_snowflake_source ‑ test_test_connection_failure
tests.unit.test_snowflake_source ‑ test_test_connection_no_warehouse
tests.unit.test_snowflake_source ‑ test_unknown_cloud_region_from_snowflake_region_id
tests.unit.test_source_helpers ‑ test_auto_browse_path_v2
tests.unit.test_source_helpers ‑ test_auto_browse_path_v2_ignores_urns_already_with
tests.unit.test_source_helpers ‑ test_auto_status_aspect
tests.unit.test_source_helpers ‑ test_auto_workunit
tests.unit.test_sql_common ‑ test_generate_foreign_key
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[awsathena://test_athena:3316/athenadb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[bigquery://test_bq:3316/bigquery]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[clickhouse://test_clickhouse:3316/clickhousedb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[druid://test_druid:1101/druiddb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[hive://test_hive:1201/hive]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[jdbc:postgres://test_redshift:5432/redshift.amazonaws]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[mongodb://test_mongodb:1201/mongo]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[mssql://test_mssql:1201/mssqldb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[mysql://test_mysql:1201/mysql]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[oracle://test_oracle:3306/oracledb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[pinot://test_pinot:3306/pinot]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[postgresql://test_postgres:5432/postgres]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[postgresql://test_redshift:5432/redshift.amazonaws]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[presto://test_presto:5432/prestodb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[redshift://test_redshift:5432/redshift]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[snowflake://test_snowflake:5432/snowflakedb]
tests.unit.test_sql_common ‑ test_get_platform_from_sqlalchemy_uri[trino://test_trino:5432/trino]
tests.unit.test_sql_common ‑ test_use_source_schema_for_foreign_key_if_not_specified
tests.unit.test_sql_utils ‑ test_guid_generators
tests.unit.test_superset_source ‑ test_default_values
tests.unit.test_superset_source ‑ test_set_display_uri
tests.unit.test_tag_urn.TestTagUrn ‑ test_invalid_urn
tests.unit.test_tag_urn.TestTagUrn ‑ test_parse_urn
tests.unit.test_transform_dataset ‑ test_add_dataset_browse_paths
tests.unit.test_transform_dataset ‑ test_add_dataset_properties
tests.unit.test_transform_dataset ‑ test_import_resolver
tests.unit.test_transform_dataset ‑ test_mark_status_dataset
tests.unit.test_transform_dataset ‑ test_mcp_add_tags_existing
tests.unit.test_transform_dataset ‑ test_mcp_add_tags_missing
tests.unit.test_transform_dataset ‑ test_mcp_multiple_transformers
tests.unit.test_transform_dataset ‑ test_mcp_multiple_transformers_replace
tests.unit.test_transform_dataset ‑ test_ownership_patching_intersect
tests.unit.test_transform_dataset ‑ test_ownership_patching_with_different_types_1
tests.unit.test_transform_dataset ‑ test_ownership_patching_with_different_types_2
tests.unit.test_transform_dataset ‑ test_ownership_patching_with_empty_mce_none_server
tests.unit.test_transform_dataset ‑ test_ownership_patching_with_empty_mce_nonempty_server
tests.unit.test_transform_dataset ‑ test_ownership_patching_with_nones
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_aspect_name
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_match
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_no_match
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_replace_existing_match
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_replace_existing_no_match
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_semantics_overwrite
tests.unit.test_transform_dataset ‑ test_pattern_add_dataset_domain_semantics_patch
tests.unit.test_transform_dataset ‑ test_pattern_dataset_ownership_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_ownership_with_invalid_type_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_ownership_with_type_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_tags_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_tags_transformation_overwrite
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_tags_transformation_patch
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_terms_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_terms_transformation_overwrite
tests.unit.test_transform_dataset ‑ test_pattern_dataset_schema_terms_transformation_patch
tests.unit.test_transform_dataset ‑ test_pattern_dataset_tags_transformation
tests.unit.test_transform_dataset ‑ test_pattern_dataset_terms_transformation
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain_aspect_name
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain_mce_support
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain_replace_existing
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain_semantics_overwrite
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_domain_semantics_patch
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_properties
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_properties_overwrite
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_properties_patch
tests.unit.test_transform_dataset ‑ test_simple_add_dataset_properties_replace_existing
tests.unit.test_transform_dataset ‑ test_simple_dataset_ownership_transformation
tests.unit.test_transform_dataset ‑ test_simple_dataset_ownership_transformer_semantics_patch
tests.unit.test_transform_dataset ‑ test_simple_dataset_ownership_with_invalid_type_transformation
tests.unit.test_transform_dataset ‑ test_simple_dataset_ownership_with_type_transformation
tests.unit.test_transform_dataset ‑ test_simple_dataset_tags_transformation
tests.unit.test_transform_dataset ‑ test_simple_dataset_terms_transformation
tests.unit.test_transform_dataset ‑ test_simple_remove_dataset_ownership
tests.unit.test_transform_dataset ‑ test_supression_works
tests.unit.test_unity_catalog_config ‑ test_only_ingest_assigned_metastore
tests.unit.test_unity_catalog_config ‑ test_profiling_requires_warehouses_id
tests.unit.test_unity_catalog_config ‑ test_within_thirty_days
tests.unit.test_unity_catalog_config ‑ test_workspace_url_should_start_with_https
tests.unit.test_urn.TestUrn ‑ test_invalid_urn
tests.unit.test_urn.TestUrn ‑ test_parse_urn
tests.unit.test_urn.TestUrn ‑ test_url_encode_urn
tests.unit.test_usage_common ‑ test_add_one_query_with_ignored_user
tests.unit.test_usage_common ‑ test_add_one_query_without_columns
tests.unit.test_usage_common ‑ test_convert_usage_aggregation_class
tests.unit.test_usage_common ‑ test_make_usage_workunit
tests.unit.test_usage_common ‑ test_make_usage_workunit_include_top_n_queries
tests.unit.test_usage_common ‑ test_multiple_query_with_ignored_user
tests.unit.test_usage_common ‑ test_multiple_query_without_columns
tests.unit.test_usage_common ‑ test_query_formatting
tests.unit.test_usage_common ‑ test_query_trimming
tests.unit.test_usage_common ‑ test_top_n_queries_validator_fails
tests.unit.test_utilities ‑ test_delayed_iter
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_get_columns_from_templated_query
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_get_columns_with_alias_and_count_star
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_get_columns_with_more_complex_join
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_get_tables_from_simple_query
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_get_tables_from_templated_query
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_tables_from_redash_query
tests.unit.test_utilities ‑ test_metadatasql_sql_parser_with_weird_lookml_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_columns_complex_query_with_union
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_columns_from_simple_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_columns_from_templated_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_columns_with_join
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_tables_from_complex_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_tables_from_simple_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_get_tables_from_templated_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_tables_from_redash_query
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_tables_with_special_names
tests.unit.test_utilities ‑ test_sqllineage_sql_parser_with_weird_lookml_query
tests.unit.test_vertica_source ‑ test_vertica_uri_https
tests.unit.test_workunit ‑ test_get_aspects_of_type_mce
tests.unit.test_workunit ‑ test_get_aspects_of_type_mcp
tests.unit.test_workunit ‑ test_get_aspects_of_type_mcpc
tests.unit.utilities.test_file_backed_collections ‑ test_custom_column[0]
tests.unit.utilities.test_file_backed_collections ‑ test_custom_column[10]
tests.unit.utilities.test_file_backed_collections ‑ test_custom_column[1]
tests.unit.utilities.test_file_backed_collections ‑ test_custom_serde
tests.unit.utilities.test_file_backed_collections ‑ test_file_cleanup
tests.unit.utilities.test_file_backed_collections ‑ test_file_dict
tests.unit.utilities.test_file_backed_collections ‑ test_file_dict_stores_counter
tests.unit.utilities.test_file_backed_collections ‑ test_file_list
tests.unit.utilities.test_file_backed_collections ‑ test_shared_connection
tests.unit.utilities.test_hive_schema_to_avro ‑ test_get_avro_schema_for_hive_column
tests.unit.utilities.test_hive_schema_to_avro ‑ test_get_avro_schema_for_struct_hive_column
tests.unit.utilities.test_hive_schema_to_avro ‑ test_get_avro_schema_for_struct_hive_with_duplicate_column
tests.unit.utilities.test_hive_schema_to_avro ‑ test_get_avro_schema_for_struct_hive_with_duplicate_column2
tests.unit.utilities.test_lossy_collections ‑ test_lossydict_sampling[10-False-14]
tests.unit.utilities.test_lossy_collections ‑ test_lossydict_sampling[100-True-1000]
tests.unit.utilities.test_lossy_collections ‑ test_lossydict_sampling[4-False-4]
tests.unit.utilities.test_lossy_collections ‑ test_lossylist_sampling[10-False]
tests.unit.utilities.test_lossy_collections ‑ test_lossylist_sampling[100-True]
tests.unit.utilities.test_lossy_collections ‑ test_lossyset_sampling[10-False]
tests.unit.utilities.test_lossy_collections ‑ test_lossyset_sampling[100-True]
tests.unit.utilities.test_memory_footprint ‑ test_total_size_with_defaultdict
tests.unit.utilities.test_memory_footprint ‑ test_total_size_with_empty_dict
tests.unit.utilities.test_memory_footprint ‑ test_total_size_with_list
tests.unit.utilities.test_memory_footprint ‑ test_total_size_with_none
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_with_reserved_chars[test&database.test(schema.test*table]
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_with_reserved_chars[test-database,test-schema,test-table]
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_with_reserved_chars[test_database,(test$schema),test+table]
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_without_reserved_chars_no_change[test&database.%testschema.test*table]
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_without_reserved_chars_no_change[test-database.test-schema.test-table]
tests.unit.utilities.test_urn_encoder ‑ test_encode_string_without_reserved_chars_no_change[test_database.test$schema.test+table]
View more details on GitHub Actions