Discussion on Datahub CLI logging behavior and configuration suggestions

ingestion
1

Original Slack Thread

Datahub version : v0.12.0
Ingestion source : ldap
Datahub CLI version : v0.9.6.4

While ingesting data from our ldap instance the datahub cli creates around 60GB of logs. In the datahub ingestion ui we can fetch and view these logs.
I have a few doubts:

  1. Where are these logs actually saved
  2. Are these logs actually cleared up
  3. Any cli configuration that could reduce the logged data size
2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)