[2023-08-04 09:26:39,769] DEBUG {datahub.telemetry.telemetry:309} - Sending telemetry for function-call
[2023-08-04 09:26:40,005] INFO {datahub.cli.ingest_cli:173} - DataHub CLI version: 0.10.5
[2023-08-04 09:26:40,123] DEBUG {datahub.ingestion.sink.datahub_rest:118} - Setting env variables to override config
[2023-08-04 09:26:40,124] DEBUG {datahub.ingestion.sink.datahub_rest:120} - Setting gms config
[2023-08-04 09:26:40,124] DEBUG {datahub.ingestion.run.pipeline:212} - Sink type datahub-rest (<class 'datahub.ingestion.sink.datahub_rest.DatahubRestSink'>) configured
[2023-08-04 09:26:40,124] INFO {datahub.ingestion.run.pipeline:213} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://datahub-gms.as-in.io> with token: eyJh**********U7Sc
[2023-08-04 09:26:40,183] DEBUG {datahub.ingestion.sink.datahub_rest:118} - Setting env variables to override config
[2023-08-04 09:26:40,183] DEBUG {datahub.ingestion.sink.datahub_rest:120} - Setting gms config
[2023-08-04 09:26:40,183] DEBUG {datahub.ingestion.reporting.datahub_ingestion_run_summary_provider:125} - Ingestion source urn = urn:li:dataHubIngestionSource:cli-05a5746e49c4efc8a3d2044d1a297d76
[2023-08-04 09:26:40,185] DEBUG {datahub.emitter.rest_emitter:260} - Attempting to emit to DataHub GMS; using curl equivalent to:
curl -X POST -H 'User-Agent: python-requests/2.28.1' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'Connection: keep-alive' -H 'X-RestLi-Protocol-Version: 2.0.0' -H 'Content-Type: application/json' -H 'Authorization: <redacted>' --data '{"proposal": {"entityType": "dataHubIngestionSource", "entityUrn":"urn:li:dataHubIngestionSource:cli-05a5746e49c4efc8a3d2044d1a297d76", "changeType": "UPSERT", "aspectName": "dataHubIngestionSourceInfo", "aspect": {"value": "{\"name\": \"[CLI] bigquery [urn:li:dataHubIngestionSource:72dc6fb3-7573-4991-91b9-068969132b08]\", \"type\": \"bigquery\", \"platform\": \"urn:li:dataPlatform:unknown\", \"config\": {\"recipe\": \"{\\\"source\\\": {\\\"type\\\": \\\"bigquery\\\", \\\"config\\\": {\\\"extract_column_lineage\\\": false, \\\"env\\\": \\\"TEST\\\", \\\"rate_limit\\\": true, \\\"requests_per_min\\\": 60, \\\"include_table_lineage\\\": true, \\\"include_usage_statistics\\\": true, \\\"include_tables\\\": true, \\\"include_views\\\": true, \\\"lineage_use_sql_parser\\\": true, \\\"incremental_lineage\\\": false, \\\"max_query_duration\\\": 1800, \\\"profiling\\\": {\\\"enabled\\\": true, \\\"profile_table_level_only\\\": false}, \\\"stateful_ingestion\\\": {\\\"enabled\\\": true}, \\\"credential\\\": {\\\"project_id\\\": \\\"aftership-test\\\", \\\"private_key\\\": \\\"********\\\", \\\"private_key_id\\\": \\\"********\\\", \\\"client_email\\\": \\\"<mailto:tssd-792@aftership-test.iam.gserviceaccount.com|tssd-792@aftership-test.iam.gserviceaccount.com>\\\", \\\"client_id\\\": \\\"101426640770366287958\\\"}, \\\"dataset_pattern\\\": {\\\"allow\\\": [\\\"hods_utc\\\", \\\"ods_utc\\\"]}}}, \\\"pipeline_name\\\": \\\"urn:li:dataHubIngestionSource:72dc6fb3-7573-4991-91b9-068969132b08\\\"}\", \"version\": \"0.10.5\", \"executorId\": \"__datahub_cli_\"}}", "contentType": "application/json"}}}' '<http://datahub-gms.as-in.io/aspects?action=ingestProposal>'
[2023-08-04 09:26:40,243] DEBUG {datahub.ingestion.run.pipeline:287} - Reporter type:datahub,<class 'datahub.ingestion.reporting.datahub_ingestion_run_summary_provider.DatahubIngestionRunSummaryProvider'> configured.
[2023-08-04 09:26:43,577] WARNING {py.warnings:109} - /usr/local/lib/python3.8/dist-packages/ratelimiter.py:127: DeprecationWarning: "@coroutine" decoratoris deprecated since Python 3.8, use "async def" instead
__aexit__ = asyncio.coroutine(__exit__)
[2023-08-04 09:26:43,694] WARNING {py.warnings:109} - /usr/local/lib/python3.8/dist-packages/datahub/ingestion/source/bigquery_v2/bigquery_config.py:210: ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.
super().__init__(**data)
[2023-08-04 09:26:43,696] WARNING {datahub.ingestion.source.bigquery_v2.bigquery_config:268} - Please update `dataset_pattern` to match against fully qualified schema name `<project_id>.<dataset_name>` and set config `match_fully_qualified_names : True`.Current default `match_fully_qualified_names: False` is only to maintain backward compatibility. The config option `match_fully_qualified_names` will be deprecated in future and the default behavior will assume `match_fully_qualified_names: True`.
[2023-08-04 09:26:43,697] DEBUG {datahub.telemetry.telemetry:309} - Sending telemetry for function-call
[2023-08-04 09:26:43,938] DEBUG {datahub.entrypoints:196} - Error: Failed to configure the source (bigquery): 1 validation error for BigQueryV2Config
extract_column_lineage
extra fields not permitted (type=value_error.extra)
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/run/pipeline.py", line 120, in _add_init_error_context
yield
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/run/pipeline.py", line 226, in __init__
self.source = source_class.create(
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 263, in create
config = BigQueryV2Config.parse_obj(config_dict)
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/source/bigquery_v2/bigquery_config.py", line 210, in __init__
super().__init__(**data)
File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for BigQueryV2Config
extract_column_lineage
extra fields not permitted (type=value_error.extra)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/datahub/entrypoints.py", line 186, in main
sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/datahub/telemetry/telemetry.py", line 448, in wrapper
raise e
File "/usr/local/lib/python3.8/dist-packages/datahub/telemetry/telemetry.py", line 397, in wrapper
res = func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
return func(ctx, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/datahub/cli/ingest_cli.py", line 187, in run
pipeline = Pipeline.create(
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/run/pipeline.py", line 336, in create
return cls(
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/run/pipeline.py", line 230, in __init__
<http://logger.info|logger.info>("Source configured successfully.")
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/dist-packages/datahub/ingestion/run/pipeline.py", line 122, in _add_init_error_context
raise PipelineInitError(f"Failed to {step}: {e}") from e
datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (bigquery): 1 validation error for BigQueryV2Config
extract_column_lineage
extra fields not permitted (type=value_error.extra)
Failed to configure the source (bigquery): 1 validation error for BigQueryV2Config
extract_column_lineage
extra fields not permitted (type=value_error.extra)
[2023-08-04 09:26:43,941] DEBUG {datahub.entrypoints:201} - DataHub CLI version: 0.10.5 at /usr/local/lib/python3.8/dist-packages/datahub/__init__.py
[2023-08-04 09:26:43,941] DEBUG {datahub.entrypoints:204} - Python version: 3.8.0 (default, Dec 9 2021, 17:53:27)
[GCC 8.4.0] at /usr/bin/python3.8 on Linux-5.4.0-1103-gcp-x86_64-with-glibc2.27
[2023-08-04 09:26:43,941] DEBUG {datahub.entrypoints:209} - GMS config {'models': {}, 'patchCapable': True, 'versions': {'linkedin/datahub': {'version': 'v0.10.5', 'commit': '4f9fc671dcea03cbb22b7c0e02e29bdf88ba955f'}}, 'managedIngestion': {'defaultCliVersion': '@cliMajorVersion@', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'timeZone': 'GMT', 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'datahub': {'serverType': 'quickstart'}, 'noCode': 'true'}```
Here is log from local machine cc <@U02AF5P6QDS>
I run in debug model