Troubleshooting Redshift Ingestion Stuck at "Pending" Status

troubleshoot
1

Original Slack Thread

Hello! We just upgraded from 10.1 to 10.4 after getting some errors with our redshift ingestion. We were able to upgrade but we are getting a different error when running our ingestions.

Current Version: 10.4
Deployment type: Kubernetes with Helm
UI or CLI: UI

Issue:
When running a redshift ingestion manually the request is stuck in “pending” and never executes.

Looking at pod logs I see that the datahub-actions pod is giving us this error:

[2024-03-28 17:08:33,876] INFO     {datahub_actions.cli.actions:98} - Action Pipeline with name 'ingestion_executor' is now running.
Exception in thread Thread-1 (run_pipeline):
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/deserializing_consumer.py", line 137, in poll
    value = self._value_deserializer(value, ctx)
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/schema_registry/avro.py", line 324, in __call__
    schema = self._registry.get_schema(schema_id)
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 368, in get_schema
    response = self._rest_client.get('schemas/ids/{}'.format(schema_id))
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 124, in get
    return self.send_request(url, method='GET', query=query)
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 174, in send_request
    raise SchemaRegistryError(response.status_code,
confluent_kafka.schema_registry.error.SchemaRegistryError: Unauthorized to perform this action. (HTTP status code 401, SR code None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/site-packages/datahub_actions/pipeline/pipeline_manager.py", line 42, in run_pipeline
    pipeline.run()
  File "/usr/local/lib/python3.10/site-packages/datahub_actions/pipeline/pipeline.py", line 161, in run
    for enveloped_event in enveloped_events:
  File "/usr/local/lib/python3.10/site-packages/datahub_actions/plugin/source/kafka/kafka_event_source.py", line 152, in events
    msg = self.consumer.poll(timeout=2.0)
  File "/usr/local/lib/python3.10/site-packages/confluent_kafka/deserializing_consumer.py", line 139, in poll
    raise ValueDeserializationError(exception=se, kafka_message=msg)
confluent_kafka.error.ValueDeserializationError: KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unauthorized to perform this action. (HTTP status code 401, SR code None)"}```
It looks permissions related but no idea where to start.![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06RECNKFAT/image.png)
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!