Uploading All Metadata Fields at Once Using Java Client RestEmitter in DataHub

ingestion
1

Original Slack Thread

<@U06TM7M40RK> Hi,How can I upload all metadata fields at once using the Java client RestEmitter provided by DataHub?

2

To upload all metadata fields at once using the Java client RestEmitter provided by DataHub, you can utilize the MetadataChangeProposalWrapper to construct a metadata change proposal and then emit it using the RestEmitter. Here’s a summarized approach based on the available data sources:

  1. Create a RestEmitter: You can create a RestEmitter using the RestEmitter.createWithDefaults() method or customize it using a builder pattern to specify server details and other configurations.

  2. Construct a MetadataChangeProposalWrapper: This involves specifying the entity type, entity URN, and the aspect you want to update. You can set multiple aspects if needed.

  3. Emit the Metadata: Use the emit method of the RestEmitter to send the metadata change proposal to DataHub. You can choose between a blocking call using a Future or a non-blocking call using a Callback.

Here is a basic example of how you might implement this:

import com.linkedin.dataset.DatasetProperties;
import datahub.client.rest.RestEmitter;
import datahub.event.MetadataChangeProposalWrapper;
import java.util.concurrent.Future;
import datahub.client.MetadataWriteResponse;

// Create the emitter
RestEmitter emitter = RestEmitter.createWithDefaults();

// Construct the metadata change proposal
MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
    .entityType("dataset")
    .entityUrn("urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my-dataset.user-table,PROD)")
    .upsert()
    .aspect(new DatasetProperties().setDescription("This is the canonical User profile dataset"))
    .build();

// Emit the metadata
Future<MetadataWriteResponse> requestFuture = emitter.emit(mcpw, null);

This example demonstrates how to set up a RestEmitter, construct a MetadataChangeProposalWrapper with a dataset description, and emit it to DataHub. You can add more aspects to the MetadataChangeProposalWrapper as needed to upload all metadata fields at once.

Would you like more detailed information or further examples?

Sources:

3

我使用这个方案时一直失败，每次提示Transfer-Encoding and Content-Length，请解释原因

4

<@U06TM7M40RK> 我使用这个方案时一直失败，每次提示Transfer-Encoding and Content-Length，请解释原因

5

I’m not able to find any data sources to answer your question. You might want to provide details about the specific error message or logs you are encountering. If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

6

<@U06TM7M40RK> 我使用以下代码，把所有元数据的字段都上传到datahub，每次都是失败，提示Transfer-Encoding and Content-Length
代码如下“

RestEmitter emitter = RestEmitter.createWithDefaults();

// Construct the metadata change proposal
MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
    .entityType("dataset")
    .entityUrn("urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my-dataset.user-table,PROD)")
    .upsert()
    .aspect(new DatasetProperties().setDescription("This is the canonical User profile dataset"))
    .build();

// Emit the metadata
Future&lt;MetadataWriteResponse&gt; requestFuture = emitter.emit(mcpw, null);```
”
7

I’m not able to find any data sources to answer your question. You might want to provide details about the server configuration or any custom headers being used. If this doesn’t help, please reach out to the DataHub support team for further assistance. :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.

8

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><