I am getting below error when trying to ingest metadata from Redshift using UI connector.I have granted below permission as well.
ALTER USER datahub_user WITH SYSLOG ACCESS UNRESTRICTED;
GRANT SELECT ON pg_catalog.svv_table_info to datahub_user;
GRANT SELECT ON pg_catalog.svl_user_info to datahub_user;
[6:43 PM] redshift_connector.error.ProgrammingError: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2948', 'R': 'aclcheck_error'}
Could you please share ingestion debug logs
Execution finished with errors.
{'exec_id': '063b3332-5e3b-4742-b1dd-2f7437043a74',
'infos': ['2023-09-20 10:31:44.835196 INFO: Starting execution for task with name=RUN_INGEST',
"2023-09-20 10:31:51.139639 INFO: Failed to execute 'datahub ingest'",
'2023-09-20 10:31:51.140664 INFO: Caught exception EXECUTING task_id=063b3332-5e3b-4742-b1dd-2f7437043a74, name=RUN_INGEST, '
'stacktrace=Traceback (most recent call last):\n'
' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
' task_event_loop.run_until_complete(task_future)\n'
' File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
' return future.result()\n'
' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 231, in execute\n'
' raise TaskError("Failed to execute \'datahub ingest\'")\n'
"acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"],
'errors': []}
~~~~ Ingestion Report ~~~~
{
"cli": {
"cli_version": "0.10.4.2",
"cli_entry_location": "/usr/local/lib/python3.10/site-packages/datahub/__init__.py",
"py_version": "3.10.11 (main, May 23 2023, 09:22:32) [GCC 10.2.1 20210110]",
"py_exec_path": "/usr/local/bin/python",
"os_details": "Linux-5.15.49-linuxkit-pr-aarch64-with-glibc2.31",
"peak_memory_usage": "314.77 MB",
"mem_info": "314.77 MB",
"peak_disk_usage": "13.46 GB",
"disk_info": {
"total": "62.67 GB",
"used": "13.46 GB",
"free": "46 GB"
}
},
"source": {
"type": "redshift",
"report": {
"events_produced": 0,
"events_produced_per_sec": 0,
"entities": {},
"aspects": {},
"warnings": {},
"failures": {},
"soft_deleted_stale_entities": [],
"tables_scanned": 0,
"views_scanned": 0,
"entities_profiled": 0,
"filtered": [],
"profiling_skipped_not_updated": {},
"profiling_skipped_size_limit": {},
"profiling_skipped_row_limit": {},
"num_tables_not_eligible_profiling": {},
"upstream_lineage": {},
"usage_extraction_sec": {},
"lineage_extraction_sec": {},
"table_processed": {},
"table_filtered": {},
"view_filtered": {},
"view_processed": {},
"table_cached": {},
"view_cached": {},
"metadata_extraction_sec": {},
"operational_metadata_extraction_sec": {},
"lineage_mem_size": {},
"tables_in_mem_size": {},
"views_in_mem_size": {},
"num_operational_stats_skipped": 0,
"num_usage_stat_skipped": 0,
"num_lineage_tables_dropped": 0,
"num_lineage_dropped_query_parser": 0,
"num_lineage_dropped_not_support_copy_path": 0,
"start_time": "2023-09-20 10:31:49.383294 (now)",
"running_time": "0.37 seconds"
}
},
"sink": {
"type": "datahub-rest",
"report": {
"total_records_written": 0,
"records_written_per_second": 0,
"warnings": [],
"failures": [],
"start_time": "2023-09-20 10:31:46.515180 (3.24 seconds ago)",
"current_time": "2023-09-20 10:31:49.753192 (now)",
"total_duration_in_seconds": 3.24,
"gms_version": "null",
"pending_requests": 0
}
}
}
~~~~ Ingestion Logs ~~~~
Obtaining venv creation lock...
Acquired venv creation lock
venv setup time = 0
This version of datahub supports report-to functionality
datahub ingest run -c /tmp/datahub/ingest/063b3332-5e3b-4742-b1dd-2f7437043a74/recipe.yml --report-to /tmp/datahub/ingest/063b3332-5e3b-4742-b1dd-2f7437043a74/ingestion_report.json
[2023-09-20 10:31:46,455] INFO {datahub.cli.ingest_cli:173} - DataHub CLI version: 0.10.4.2
[2023-09-20 10:31:46,517] INFO {datahub.ingestion.run.pipeline:213} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://datahub-gms:8080>
Please update `schema_pattern` to match against fully qualified schema name `<database_name>.<schema_name>` and set config `match_fully_qualified_names : True`.Current default `match_fully_qualified_names: False` is only to maintain backward compatibility. The config option `match_fully_qualified_names` will be deprecated in future and the default behavior will assume `match_fully_qualified_names: True`.
[2023-09-20 10:31:49,383] INFO {datahub.ingestion.run.pipeline:230} - Source configured successfully.
[2023-09-20 10:31:49,384] INFO {datahub.cli.ingest_cli:129} - Starting metadata ingestion
[2023-09-20 10:31:49,634] WARNING {py.warnings:109} - /usr/local/lib/python3.10/site-packages/redshift_connector/core.py:603: DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated.
ssl_context: SSLContext = SSLContext()
[2023-09-20 10:31:49,634] WARNING {py.warnings:109} - /usr/local/lib/python3.10/site-packages/redshift_connector/core.py:603: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
ssl_context: SSLContext = SSLContext()
[2023-09-20 10:31:49,722] INFO {datahub.ingestion.source.redshift.redshift:366} - Processing db dev with name dev
[2023-09-20 10:31:49,753] INFO {datahub.ingestion.reporting.file_reporter:52} - Wrote UNKNOWN report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/063b3332-5e3b-4742-b1dd-2f7437043a74/ingestion_report.json' mode='w' encoding='UTF-8'>
[2023-09-20 10:31:49,753] INFO {datahub.cli.ingest_cli:134} - Source (redshift) report:
{'events_produced': 0,
'events_produced_per_sec': 0,
'entities': {},
'aspects': {},
'warnings': {},
'failures': {},
'soft_deleted_stale_entities': [],
'tables_scanned': 0,
'views_scanned': 0,
'entities_profiled': 0,
'filtered': [],
'profiling_skipped_not_updated': {},
'profiling_skipped_size_limit': {},
'profiling_skipped_row_limit': {},
'num_tables_not_eligible_profiling': {},
'upstream_lineage': {},
'usage_extraction_sec': {},
'lineage_extraction_sec': {},
'table_processed': {},
'table_filtered': {},
'view_filtered': {},
'view_processed': {},
'table_cached': {},
'view_cached': {},
'metadata_extraction_sec': {},
'operational_metadata_extraction_sec': {},
'lineage_mem_size': {},
'tables_in_mem_size': {},
'views_in_mem_size': {},
'num_operational_stats_skipped': 0,
'num_usage_stat_skipped': 0,
'num_lineage_tables_dropped': 0,
'num_lineage_dropped_query_parser': 0,
'num_lineage_dropped_not_support_copy_path': 0,
'start_time': '2023-09-20 10:31:49.383294 (now)',
'running_time': '0.37 seconds'}
[2023-09-20 10:31:49,754] INFO {datahub.cli.ingest_cli:137} - Sink (datahub-rest) report:
{'total_records_written': 0,
'records_written_per_second': 0,
'warnings': [],
'failures': [],
'start_time': '2023-09-20 10:31:46.515180 (3.24 seconds ago)',
'current_time': '2023-09-20 10:31:49.754016 (now)',
'total_duration_in_seconds': 3.24,
'gms_version': 'null',
'pending_requests': 0}
[2023-09-20 10:31:50,034] ERROR {datahub.entrypoints:199} - Command failed: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2948', 'R': 'aclcheck_error'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/datahub/entrypoints.py", line 186, in main
sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 448, in wrapper
raise e
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 397, in wrapper
res = func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
return func(ctx, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 198, in run
loop.run_until_complete(run_func_check_upgrade(pipeline))
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 158, in run_func_check_upgrade
ret = await the_one_future
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 149, in run_pipeline_async
return await loop.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 140, in run_pipeline_to_completion
raise e
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 132, in run_pipeline_to_completion
pipeline.run()
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 367, in run
for wu in itertools.islice(
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 113, in auto_stale_entity_removal
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 137, in auto_workunit_reporter
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 202, in auto_browse_path_v2
for urn, batch in _batch_workunits_by_urn(stream):
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 282, in _batch_workunits_by_urn
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 150, in auto_materialize_referenced_tags
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 64, in auto_status_aspect
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift.py", line 375, in get_workunits_internal
``` File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift.py", line 767, in cache_tables_and_views
tables, views = RedshiftDataDictionary.get_tables_and_views(conn=connection)
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 168, in get_tables_and_views
enriched_table = RedshiftDataDictionary.enrich_tables(conn)
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 132, in enrich_tables
cur = RedshiftDataDictionary.get_query_result(
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 90, in get_query_result
cursor.execute(query)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/cursor.py", line 240, in execute
self._c.execute(self, operation, args)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/core.py", line 1783, in execute
self.handle_messages(cursor)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/core.py", line 1969, in handle_messages
raise self.error
redshift_connector.error.ProgrammingError: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2948', 'R': 'aclcheck_error'}```<@U05SWV7C160> Could you please share debug logs
I am running this through UI ingestion. Can you please guide how I can provide debug logs with you
On ingestion Finish tab there is advance section, click on that and see debug flag
Execution finished with errors.
{'exec_id': 'c29d6abf-22ea-4b78-9bd2-5715c62ba59f',
'infos': ['2023-09-21 12:21:34.140692 INFO: Starting execution for task with name=RUN_INGEST',
"2023-09-21 12:21:42.444113 INFO: Failed to execute 'datahub ingest'",
'2023-09-21 12:21:42.444985 INFO: Caught exception EXECUTING task_id=c29d6abf-22ea-4b78-9bd2-5715c62ba59f, name=RUN_INGEST, '
'stacktrace=Traceback (most recent call last):\n'
' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n'
' task_event_loop.run_until_complete(task_future)\n'
' File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
' return future.result()\n'
' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 231, in execute\n'
' raise TaskError("Failed to execute \'datahub ingest\'")\n'
"acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"],
'errors': []}
~~~~ Ingestion Report ~~~~
{
"cli": {
"cli_version": "0.10.4.2",
"cli_entry_location": "/usr/local/lib/python3.10/site-packages/datahub/__init__.py",
"py_version": "3.10.11 (main, May 23 2023, 09:22:32) [GCC 10.2.1 20210110]",
"py_exec_path": "/usr/local/bin/python",
"os_details": "Linux-5.15.49-linuxkit-pr-aarch64-with-glibc2.31",
"peak_memory_usage": "315.07 MB",
"mem_info": "315.07 MB",
"peak_disk_usage": "13.71 GB",
"disk_info": {
"total": "62.67 GB",
"used": "13.71 GB",
"free": "45.74 GB"
}
},
"source": {
"type": "redshift",
"report": {
"events_produced": 0,
"events_produced_per_sec": 0,
"entities": {},
"aspects": {},
"warnings": {},
"failures": {},
"soft_deleted_stale_entities": [],
"tables_scanned": 0,
"views_scanned": 0,
"entities_profiled": 0,
"filtered": [],
"profiling_skipped_not_updated": {},
"profiling_skipped_size_limit": {},
"profiling_skipped_row_limit": {},
"num_tables_not_eligible_profiling": {},
"upstream_lineage": {},
"usage_extraction_sec": {},
"lineage_extraction_sec": {},
"table_processed": {},
"table_filtered": {},
"view_filtered": {},
"view_processed": {},
"table_cached": {},
"view_cached": {},
"metadata_extraction_sec": {},
"operational_metadata_extraction_sec": {},
"lineage_mem_size": {},
"tables_in_mem_size": {},
"views_in_mem_size": {},
"num_operational_stats_skipped": 0,
"num_usage_stat_skipped": 0,
"num_lineage_tables_dropped": 0,
"num_lineage_dropped_query_parser": 0,
"num_lineage_dropped_not_support_copy_path": 0,
"start_time": "2023-09-21 12:21:38.374970 (now)",
"running_time": "0.98 seconds"
}
},
"sink": {
"type": "datahub-rest",
"report": {
"total_records_written": 0,
"records_written_per_second": 0,
"warnings": [],
"failures": [],
"start_time": "2023-09-21 12:21:35.915174 (3.44 seconds ago)",
"current_time": "2023-09-21 12:21:39.358512 (now)",
"total_duration_in_seconds": 3.44,
"gms_version": "null",
"pending_requests": 0
}
}
}
~~~~ Ingestion Logs ~~~~
Obtaining venv creation lock...
Acquired venv creation lock
venv setup time = 0
This version of datahub supports report-to functionality
datahub --debug ingest run -c /tmp/datahub/ingest/c29d6abf-22ea-4b78-9bd2-5715c62ba59f/recipe.yml --report-to /tmp/datahub/ingest/c29d6abf-22ea-4b78-9bd2-5715c62ba59f/ingestion_report.json
[2023-09-21 12:21:34,944] DEBUG {datahub.telemetry.telemetry:280} - Sending init Telemetry
[2023-09-21 12:21:35,580] DEBUG {datahub.telemetry.telemetry:309} - Sending telemetry for function-call
[2023-09-21 12:21:35,854] INFO {datahub.cli.ingest_cli:173} - DataHub CLI version: 0.10.4.2
[2023-09-21 12:21:35,917] DEBUG {datahub.ingestion.sink.datahub_rest:118} - Setting env variables to override config
[2023-09-21 12:21:35,917] DEBUG {datahub.ingestion.sink.datahub_rest:120} - Setting gms config
[2023-09-21 12:21:35,917] DEBUG {datahub.ingestion.run.pipeline:212} - Sink type datahub-rest (<class 'datahub.ingestion.sink.datahub_rest.DatahubRestSink'>) configured
[2023-09-21 12:21:35,917] INFO {datahub.ingestion.run.pipeline:213} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://datahub-gms:8080>
[2023-09-21 12:21:35,928] DEBUG {datahub.ingestion.run.pipeline:287} - Reporter type:file,<class 'datahub.ingestion.reporting.file_reporter.FileReporter'> configured.
Please update `schema_pattern` to match against fully qualified schema name `<database_name>.<schema_name>` and set config `match_fully_qualified_names : True`.Current default `match_fully_qualified_names: False` is only to maintain backward compatibility. The config option `match_fully_qualified_names` will be deprecated in future and the default behavior will assume `match_fully_qualified_names: True`.
[2023-09-21 12:21:38,374] DEBUG {datahub.ingestion.source.state.stateful_ingestion_base:252} - Successfully created datahub state provider.
[2023-09-21 12:21:38,374] DEBUG {datahub.ingestion.run.pipeline:229} - Source type redshift (<class 'datahub.ingestion.source.redshift.redshift.RedshiftSource'>) configured
[2023-09-21 12:21:38,375] INFO {datahub.ingestion.run.pipeline:230} - Source configured successfully.
[2023-09-21 12:21:38,375] INFO {datahub.cli.ingest_cli:129} - Starting metadata ingestion
[2023-09-21 12:21:38,717] WARNING {py.warnings:109} - /usr/local/lib/python3.10/site-packages/redshift_connector/core.py:603: DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated.
ssl_context: SSLContext = SSLContext()
[2023-09-21 12:21:38,717] WARNING {py.warnings:109} - /usr/local/lib/python3.10/site-packages/redshift_connector/core.py:603: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
ssl_context: SSLContext = SSLContext()
[2023-09-21 12:21:39,173] INFO {datahub.ingestion.source.redshift.redshift:366} - Processing db dev with name dev
[2023-09-21 12:21:39,177] DEBUG {datahub.ingestion.source.redshift.redshift_schema:89} - Query :
select
database,
schema,
"table",
size,
tbl_rows,
estimated_visible_rows,
skew_rows,
last_accessed
from
pg_catalog.svv_table_info as ti
left join (
select
tbl,
max(endtime) as last_accessed
from
pg_catalog.stl_insert
group by
tbl) as la on
(la.tbl = ti.table_id)
;```[2023-09-21 12:21:39,361] INFO {datahub.ingestion.reporting.file_reporter:52} - Wrote UNKNOWN report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/c29d6abf-22ea-4b78-9bd2-5715c62ba59f/ingestion_report.json' mode='w' encoding='UTF-8'>
[2023-09-21 12:21:39,363] INFO {datahub.cli.ingest_cli:134} - Source (redshift) report:
{'events_produced': 0,
'events_produced_per_sec': 0,
'entities': {},
'aspects': {},
'warnings': {},
'failures': {},
'soft_deleted_stale_entities': [],
'tables_scanned': 0,
'views_scanned': 0,
'entities_profiled': 0,
'filtered': [],
'profiling_skipped_not_updated': {},
'profiling_skipped_size_limit': {},
'profiling_skipped_row_limit': {},
'num_tables_not_eligible_profiling': {},
'upstream_lineage': {},
'usage_extraction_sec': {},
'lineage_extraction_sec': {},
'table_processed': {},
'table_filtered': {},
'view_filtered': {},
'view_processed': {},
'table_cached': {},
'view_cached': {},
'metadata_extraction_sec': {},
'operational_metadata_extraction_sec': {},
'lineage_mem_size': {},
'tables_in_mem_size': {},
'views_in_mem_size': {},
'num_operational_stats_skipped': 0,
'num_usage_stat_skipped': 0,
'num_lineage_tables_dropped': 0,
'num_lineage_dropped_query_parser': 0,
'num_lineage_dropped_not_support_copy_path': 0,
'start_time': '2023-09-21 12:21:38.374970 (now)',
'running_time': '0.99 seconds'}
[2023-09-21 12:21:39,364] INFO {datahub.cli.ingest_cli:137} - Sink (datahub-rest) report:
{'total_records_written': 0,
'records_written_per_second': 0,
'warnings': [],
'failures': [],
'start_time': '2023-09-21 12:21:35.915174 (3.45 seconds ago)',
'current_time': '2023-09-21 12:21:39.364152 (now)',
'total_duration_in_seconds': 3.45,
'gms_version': 'null',
'pending_requests': 0}
[2023-09-21 12:21:39,365] DEBUG {datahub.telemetry.telemetry:309} - Sending telemetry for function-call
[2023-09-21 12:21:39,665] ERROR {datahub.entrypoints:199} - Command failed: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2948', 'R': 'aclcheck_error'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/datahub/entrypoints.py", line 186, in main
sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 448, in wrapper
raise e
File "/usr/local/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 397, in wrapper
res = func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper
return func(ctx, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 198, in run
loop.run_until_complete(run_func_check_upgrade(pipeline))
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 158, in run_func_check_upgrade
ret = await the_one_future
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 149, in run_pipeline_async
return await loop.run_in_executor(
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 140, in run_pipeline_to_completion
raise e
File "/usr/local/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 132, in run_pipeline_to_completion
pipeline.run()
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 367, in run
for wu in itertools.islice(
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 113, in auto_stale_entity_removal
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 137, in auto_workunit_reporter
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 202, in auto_browse_path_v2
for urn, batch in _batch_workunits_by_urn(stream):
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 282, in _batch_workunits_by_urn
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 150, in auto_materialize_referenced_tags
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 64, in auto_status_aspect
for wu in stream:
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift.py", line 375, in get_workunits_internal
self.cache_tables_and_views(connection, database)
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift.py", line 767, in cache_tables_and_views
tables, views = RedshiftDataDictionary.get_tables_and_views(conn=connection)
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 168, in get_tables_and_views
enriched_table = RedshiftDataDictionary.enrich_tables(conn)
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 132, in enrich_tables
cur = RedshiftDataDictionary.get_query_result(
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/redshift/redshift_schema.py", line 90, in get_query_result
cursor.execute(query)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/cursor.py", line 240, in execute
self._c.execute(self, operation, args)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/core.py", line 1783, in execute
self.handle_messages(cursor)
File "/usr/local/lib/python3.10/site-packages/redshift_connector/core.py", line 1969, in handle_messages
raise self.error
redshift_connector.error.ProgrammingError: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2948', 'R': 'aclcheck_error'}
[2023-09-21 12:21:39,675] DEBUG {datahub.entrypoints:201} - DataHub CLI version: 0.10.4.2 at /usr/local/lib/python3.10/site-packages/datahub/__init__.py
[2023-09-21 12:21:39,675] DEBUG {datahub.entrypoints:204} - Python version: 3.10.11 (main, May 23 2023, 09:22:32) [GCC 10.2.1 20210110] at /usr/local/bin/python on Linux-5.15.49-linuxkit-pr-aarch64-with-glibc2.31
[2023-09-21 12:21:39,675] DEBUG {datahub.entrypoints:209} - GMS config {'models': {}, 'patchCapable': True, 'versions': {'linkedin/datahub': {'version': 'null', 'commit': '7dd6e09ac5afce8bc354b68fdb9739d04f22a6a2'}}, 'managedIngestion': {'defaultCliVersion': '@cliMajorVersion@', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'timeZone': 'GMT', 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'datahub': {'serverType': 'quickstart'}, 'noCode': 'true'}```<@U05SWV7C160> Is there any issue in attaching debug log as file ? I want to see the query connector is sending to source
that particular query is failing and hence you are getting this permission error
Pleese find the attached log here
<@U05SWV7C160>
Below query is failing. Could you please check if you can execute the below query using same credential from Redshift query editor
database,
schema,
"table",
size,
tbl_rows,
estimated_visible_rows,
skew_rows,
last_accessed
from
pg_catalog.svv_table_info as ti
left join (
select
tbl,
max(endtime) as last_accessed
from
pg_catalog.stl_insert
group by
tbl) as la on
(la.tbl = ti.table_id)
;
```I am getting below error in Redshift query editor. I used super user credentials as well ERROR: permission denied for relation stl_insert [ErrorId: 1-650dbadd-5ad0271c7b7c6ac13409dc56]
Please contact your Admin, the query should work in Redshift query editor with same user/credential
Was there a solution found for this? I’m running into the same issue.
I also tried querying the pg_catalog schema and didn’t see the stl_insert table.
WHERE table_schema='pg_catalog'
AND table_name='stl_insert';```It’s because stl_insert isn’t queryable for Redshift Serverless.