Enabling Retries for DataHub Actions Executor

ingestion
1

Original Slack Thread

<@U06TM7M40RK>, I want to enable retries for datahub actions executor. How can I configure this?

2

To enable and configure retries for the DataHub actions executor, you need to set up the retry mechanism in your action configuration file. Here are the steps to achieve this:

  1. Create or Update the Configuration File: Ensure you have a configuration file (e.g., action_config.yaml) for your DataHub actions. This file should include the necessary configurations for the action executor.

  2. Configure Retry Settings: In the configuration file, you can specify the retry settings under the options section. Here is an example configuration:

    # Required: Action Pipeline Name
name: "executor_action"

# Required: Event Source - Where to source event from.
source:
  type: "kafka"
  config:
    connection:
      bootstrap: ${KAFKA_BOOTSTRAP_SERVER:-localhost:9092}
      schema_registry_url: ${SCHEMA_REGISTRY_URL:-http://localhost:8081}

# Action - What action to take on events.
action:
  type: "executor"
  config:
    # Action-specific configs (map)
    key: "value"

# Optional: DataHub API configuration
datahub:
  server: "http://localhost:8080"
  token: "your-access-token"  # Required if Metadata Service Auth enabled

# Optional: Additional pipeline options (error handling, etc)
options:
  retry_count: 3  # Number of retry attempts
  failure_mode: "CONTINUE"  # Behavior on failure
  failed_events_dir: "/tmp/datahub/actions"  # Directory to store failed events

  3. Set Up Environment Variables: Ensure that the necessary environment variables are set, such as KAFKA_BOOTSTRAP_SERVER and SCHEMA_REGISTRY_URL.

  4. Deploy the Configuration: If you are using Docker Compose, ensure that the configuration file is mounted into the datahub-actions container. For Kubernetes, you can use a ConfigMap to mount the configuration file.

  5. Run the Action: Once the configuration is deployed, you can run the action using the DataHub CLI:

    datahub actions -c /etc/datahub/actions/action_config.yaml

This setup will configure the executor to retry failed actions up to 3 times, with the specified behavior on failure.

Would you like more detailed steps on any specific part of this process?

Sources:

3

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><