Troubleshooting 'datacatalog-elasticsearch-setup-job' Errors

troubleshoot
1

Original Slack Thread

Hello guys,
I am having problem with the datacatalog-elasticsearch-setup-job job. Can anyone help me?
I have the following helm values on my deployment file:

  enabled: true

global:
  graph_service_impl: elasticsearch
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: false

  elasticsearch:
    host: "host-address"
    port: "9243"
    useSSL: true
    auth:
      username: prod_app_datahub
      password:
        secretRef: datahub-custom-secrets
        secretKey: ELASTICSEARCH_DATAHUB_PASSWORD
    indexPrefix: "datahub__"
    index:
      upgrade:
        cloneIndices: false```
But I am encountering the following error on the setup-job:
```2024/05/17 09:49:56 Waiting for: <https://host-address:9243>
2024/05/17 09:49:56 Received 200 from <https://host-address:9243>
going to use protocol: https
going to use elastic headers based on username and password
going to use prefix: 'datahub___'

 datahub_analytics_enabled: true

&gt;&gt;&gt; GET _ilm/policy/datahub___datahub_usage_event_policy response code is 404
&gt;&gt;&gt; creating _ilm/policy/datahub___datahub_usage_event_policy because it doesn't exist ...
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"
          }
        }
      }
    }
  }
}
{"statusCode":404,"error":"Not Found","message":"Not Found"}
&gt;&gt;&gt; GET _index_template/datahub___datahub_usage_event_index_template response code is 404
&gt;&gt;&gt; creating _index_template/datahub___datahub_usage_event_index_template because it doesn't exist ...
{
  "index_patterns": ["*datahub___datahub_usage_event*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "type": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        },
        "userAgent": {
          "type": "keyword"
        },
        "browserId": {
          "type": "keyword"
        }
      }
    },
    "settings": {
      "index.lifecycle.name": "datahub___datahub_usage_event_policy",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1
    }
  }
}{"statusCode":404,"error":"Not Found","message":"Not Found"}
&gt;&gt;&gt; GET _data_stream/datahub___datahub_usage_event response code is 404
&gt;&gt;&gt; creating _data_stream/datahub___datahub_usage_event because it doesn't exist ...
sed: /index/usage-event/datahub_usage_event: No such file or directory
2024/05/17 09:49:57 Command finished successfully.
{"statusCode":404,"error":"Not Found","message":"Not Found"}```
Thanks !
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

Datahub Application Version: 0.13.2
Datahub Chart Version: 0.4.10

4

The above error I think was due to a wrong endpoint address. I took the Kibana endpoint instead of the ES endpoint. But even fixing it, I am having the following error:

2024/05/17 13:31:03 Received 200 from <https://es-endpoint:9243>
going to use protocol: https
going to use elastic headers based on username and password
going to use prefix: 'datahub-_'

 datahub_analytics_enabled: true

&gt;&gt;&gt; GET _ilm/policy/datahub-_datahub_usage_event_policy response code is 404
&gt;&gt;&gt; creating _ilm/policy/datahub-_datahub_usage_event_policy because it doesn't exist ...
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"
          }
        }
      }
    }
  }
}
{"acknowledged":true}
&gt;&gt;&gt; GET _index_template/datahub-_datahub_usage_event_index_template response code is 404
&gt;&gt;&gt; creating _index_template/datahub-_datahub_usage_event_index_template because it doesn't exist ...
{
  "index_patterns": ["*datahub-_datahub_usage_event*"],
  "data_stream": { },
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "type": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        },
        "userAgent": {
          "type": "keyword"
        },
        "browserId": {
          "type": "keyword"
        }
      }
    },
    "settings": {
      "index.lifecycle.name": "datahub-_datahub_usage_event_policy",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1
    }
  }
}{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index template [datahub-_datahub_usage_event_index_template] has index patterns [*datahub-_datahub_usage_event*] matching patterns from existing templates [.slo-observability.sli,.slo-observability.summary] with patterns (.slo-observability.sli =&gt; [.slo-observability.sli-*],.slo-observability.summary =&gt; [.slo-observability.summary-*]) that have the same priority [500], multiple index templates may not match during index creation, please use a different priority"}],"type":"illegal_argument_exception","reason":"index template [datahub-_datahub_usage_event_index_template] has index patterns [*datahub-_datahub_usage_event*] matching patterns from existing templates [.slo-observability.sli,.slo-observability.summary] with patterns (.slo-observability.sli =&gt; [.slo-observability.sli-*],.slo-observability.summary =&gt; [.slo-observability.summary-*]) that have the same priority [500], multiple index templates may not match during index creation, please use a different priority"},"status":400}
&gt;&gt;&gt; GET _data_stream/datahub-_datahub_usage_event response code is 404
&gt;&gt;&gt; creating _data_stream/datahub-_datahub_usage_event because it doesn't exist ...
sed: /index/usage-event/datahub_usage_event: No such file or directory
2024/05/17 13:31:04 Command finished successfully.
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [datahub-_datahub_usage_event]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [datahub-_datahub_usage_event]"},"status":400}```
5

Here’s a breakdown of the error message:
• The index template datahub-_datahub_usage_event_index_template is causing the error.
• It has index patterns *datahub-_datahub_usage_event*.
• There are existing templates with patterns .slo-observability.sli and .slo-observability.summary.
• These existing templates have patterns like .slo-observability.sli-* and .slo-observability.summary-*.
• All these patterns have the same priority of 500, and multiple index templates cannot match during index creation.