Hello, I’m trying to ingest superset, but it’s only ingesting dashboards and charts. The connection between charts and postgres tables (already ingested in datahub) are not made.
Hey there!
<@U06TM7M40RK> Can you help with the questions in this thread?
Please continue to tag RunLLM (e.g <@U06TM7M40RK>) to have it continue generating responses!
To address the issue where Superset ingestion is only ingesting dashboards and charts but not making the connection between charts and Postgres tables, here are some steps and considerations:
-
Verify API Endpoints:
Ensure that your Superset instance is exposing the necessary API endpoints. The Superset connector extracts datasets by contacting the/api/v1/dataset/
endpoint. You can manually check this endpoint to see if it returns the expected data.curl -u username:password http://host:8088/api/v1/dataset/
-
Check Permissions:
Ensure that the user account used for the ingestion has the necessary permissions to access the datasets. There have been instances where permissions issues prevented the ingestion from retrieving the data. -
Enable Debug Logging:
Run the ingestion with the--debug
flag to get detailed logs. This can help identify any issues during the ingestion process.datahub --debug ingest run -c <recipe-path> &> /tmp/superset-ingest.log
-
Inspect the Payload:
Add logging to inspect the payload being sent to DataHub. This can help determine if the payload is empty or if there are any issues with the data being retrieved from Superset. -
Check Stateful Ingestion Configuration:
Ensure that your stateful ingestion configuration is correctly set up. Verify that thestateful_ingestion
settings are properly configured in your ingestion recipe.
Here is an example of an ingestion recipe for Superset with stateful ingestion enabled:
source:
type: superset
config:
connect_uri: 'http://host:8088'
username: 'username'
password: 'password'
provider: db
stateful_ingestion:
enabled: true
remove_stale_metadata: false
sink:
type: datahub-rest
config:
server: 'http://localhost:8080'
token: 'your-token'
- Known Issues and Updates:
There have been updates and discussions regarding the ingestion of datasets from Superset. For instance, a <https://github.com/datahub-project/datahub/pull/10592|GitHub pull request> introduces support for the Datasets entity in the Superset data ingestion plugin, enhancing its integration capabilities with DataHub. This might be relevant to your issue.
Would you like more detailed steps or further assistance on any specific part of the process?
Sources:
- <Slack Discussion on Superset Ingestion Issues> 0 button 0 button Hint: Mention @runllm in the thread for followups.
Hi ! Did this interaction resolve your issue or do you still require assistance ?
Please tag @ RunLLM if you still need further assistance !
Or tag @Datahub Community Support if you require a person to take a look
Hope this helps ! ><