Configuring Hive Data Ingestion with MapR Ticket or JWT Token for Data Lineage

ingestion
1

Original Slack Thread

<@U06TM7M40RK> how do I ingest tables data from Hive which authenticated with maprtkt or JWT token for data lineage

2

To ingest table data from Hive with authentication using MapR Ticket (maprtkt) or JWT token for data lineage, you will need to configure your DataHub ingestion recipe to handle the specific authentication mechanism. However, the provided data sources do not explicitly mention support for MapR Ticket or JWT token authentication in Hive ingestion. Here are some general steps and considerations based on the available information:

  1. Kerberos Authentication: If your Hive setup uses Kerberos, you can configure the Hive ingestion recipe to use Kerberos authentication. This involves setting up the krb5.conf and keytab files correctly and specifying the Kerberos details in the ingestion recipe. For example:

    source:
  type: hive
  config:
    host_port: 'hive_host:10000'
    database: your_database
    options:
      connect_args:
        auth: KERBEROS
        kerberos_service_name: hive

  2. JWT Token Authentication: The data sources do not provide specific instructions for JWT token authentication with Hive. Typically, JWT token authentication would require setting up a custom authentication mechanism or using a proxy that supports JWT tokens.

  3. MapR Ticket Authentication: Similar to JWT, the data sources do not mention MapR Ticket authentication. You may need to explore custom configurations or use a compatible JDBC driver that supports MapR Ticket authentication.

  4. Lineage Extraction: For lineage extraction, ensure that your ingestion source supports lineage capabilities. According to the <https://datahubproject.io/docs/0.14.0/generated/lineage/lineage-feature-guide|DataHub Lineage Feature Guide>, DataHub supports automatic lineage detection for certain platforms. If Hive lineage is not automatically extracted, you may need to manually configure lineage or use a tool like the DataHub Spark Agent for Spark-based lineage.

Would you like more detailed steps on any of these points or further assistance on a specific part of the process?

Sources: