Updating and Mass Changing Parameters in DataHub Ingestion Sources

ingestion
1

Original Slack Thread

<@U06TM7M40RK> Can you please give an example of a GraphQL query - updateIngestionSource

2

To update an ingestion source using the GraphQL API, you can use the updateIngestionSource mutation. This mutation requires the urn of the ingestion source you want to update and the input parameters for the update.

Here is an example of how you can structure the GraphQL query to update an ingestion source:

mutation {
  updateIngestionSource(
    urn: "urn:li:dataHubIngestionSource:example-urn",
    input: {
      name: "Updated Ingestion Source Name",
      type: "mysql",
      description: "Updated description for the ingestion source",
      schedule: {
        interval: "*/10 * * * *",
        timezone: "UTC"
      },
      config: {
        recipe: "{\"source\":{\"type\":\"mysql\",\"config\":{\"include_tables\":true,\"database\":null,\"password\":\"${MYSQL_PASSWORD}\",\"profiling\":{\"enabled\":false},\"host_port\":null,\"include_views\":true,\"username\":\"${MYSQL_USERNAME}\"}},\"pipeline_name\":\"urn:li:dataHubIngestionSource:example-urn\"}",
        version: "0.8.18",
        executorId: "updatedExecutorId",
        debugMode: false,
        extraArgs: []
      }
    }
  ) {
    urn
  }
}

In this example:

  • The urn parameter specifies the unique identifier of the ingestion source you want to update.
  • The input parameter contains the updated details for the ingestion source, including the name, type, description, schedule, and configuration.

Would you like more detailed guidance on setting up this GraphQL query or any additional information?

Sources:

3

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)

<@U06TM7M40RK> Can you help with the questions in this thread?

:information_source: We’re experimenting with a new AI assistant for questions in the <#CUMUWQU66> channel to help you track down helpful resources more quickly. Keep in mind that responses may not always be spot-on; feel free to ask it follow-up questions and don’t forget to provide its responses a :thumbsup: or :thumbsdown: so other folks in the Community know if you’re still looking for help on the topic!

4

Sure, I’d be happy to help! To assist you more effectively, could you please provide the following information:

  1. Are you using the UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g., 0.12.0)?
  3. What data source(s) are you integrating with DataHub (e.g., BigQuery)?

This information will help me provide you with the most relevant guidance and resources.

Sources:

5

<@U06TM7M40RK> Can I somehow massively change the parameters in the recipes for the incest?

6

Sure thing! I’m looking through the available data to answer your question. Hold on a second… :hourglass_flowing_sand:

7

<@U06TM7M40RK> Can I somehow massively change the parameters in the recipes for the ingest?