Hello Everyone, i’m exploring datahub capabilities, and try to set custom properties on a dataset. Looking at the official documentation at https://datahubproject.io/docs/api/datahub-apis/#datahub-api-comparison it says that custom properties on dataset can be added/removed/replaced using OpenApi. Now, if i come to the swagger exposed by my datahub instance, I cannot figure out where this rest api is located: the dataset seems only to support the POST (creation) method and as well, I cannot find the create aspect custom properties for that dataset. Anyone is able to help me finding this? Thanks
For this, you can use the python api -> https://datahubproject.io/docs/api/tutorials/custom-properties#add-and-remove-custom-properties-programmatically
<@U01GCJKA8P9> can help you answer how you can do it with the open API endpoint.
ok good. I’m in a first phase of this exploring, so currently, i’ve deployed it in GKE using your official helm chart. Of course now, building and deploying a java application would require time to setup etc. so it would be very very nice if i can do it with rest api
(GraphQL seems not supported, currently, but Open Api yes)
yeah, I can understand, we have also option to use the Python API where maybe the setup can be easier.
but my colleagues will reply to you about how you can do with the openapi
setting up another pod with the python runtime, python app and so on would require time, not a java specific problem, but I need to keep my environment reproducible, so setting up a new application requires pipelines, building, testing etc . Rest api or graphql can be called directly from my computer
<@UV14447EU> I just realized the endpoint for the GMS has already been provisioned by the helm chart, so i’m able to connect externally using java. I’m trying to modify an existing dataset i’ve imported using postgres ingestion, but I see, following the link about custom properties cannot patch that entity… prerequisites says that you should have the quickstart data (that i don’t have since i provisioned it on gke), so the question is: on a postgres dataset, can i set any custom properties (without changing the schema of that dataset) or it requires some other changes to the entity before patching? Or… alternatives… is the example working and aligned to java lib 0.11.0 ?
error is Exception in thread "main" java.lang.IllegalArgumentException: No patches specified.
at datahub.client.patch.AbstractMultiFieldPatchBuilder.buildPatch(AbstractMultiFieldPatchBuilder.java:71)
at datahub.client.patch.AbstractMultiFieldPatchBuilder.build(AbstractMultiFieldPatchBuilder.java:38)
at Main.main(Main.java:34)
so the method “build” is failing
Custom properties can be modified via the generic /entities/v1 endpoint. This endpoint is exceedingly generic but very powerful. The drawback is that it is not intuitive and we’re building out other more explicit endpoints over time. In version v0.11.0, there is a way to interact directly with the openapi entities endpoint to manipulate any aspect however patching is not supported. This means to remove/add 1 property on a list of properties, rather then setting the entire list, requires a read before write pattern unless you can readily compute all properties at once. Other SDKs rely on the http://rest.li|rest.li endpoint which include support for patch operations. Since you are familiar with java and the openapi documentation is lacking let me show you the java backend. The entities https://github.com/datahub-project/datahub/blob/master/metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/entities/EntitiesController.java#L131|endpoint is actually accepting a list of UpsertAspectRequests. This includes a https://github.com/datahub-project/datahub/blob/master/metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/dto/UpsertAspectRequest.java#L29|field called aspect which is a massive oneOf which includes datasetProperties Here is a concrete example and as mentioned it will replace the customProperties on a dataset.
"entityType": "dataset",
"entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
"aspect": {
"__type": "DatasetProperties",
"customProperties": {
"foo": "bar"
}
}
}]```