for cli ingestion, is there a way to retain all the ingestion warnings in a file after a run? I see a --report-to option that stores the summary, but i don’t see an option for ALL the individual warnings. the failure_log option only keeps errors in ingestion.
To clarify, the failure_log gets used when the sink fails to write.
Warnings and failures in the report are collected in an in-memory data structure, and generally use our LossyList and LossyDict types because we previously ran into issues where the size of the report would blow up and cause OOMs
We don’t have a good out of the box way to store all warnings - if you’ve got lots of memory, you can probably monkeypatch those lossy data structures to have large size limits and therefore retain more items, but it’s not ideal
All that said - if there’s a specific report field where you’d like to see more items, we can probably tweak the defaults so those fields retain say 25 items instead of the default of 10
thanks Harshal!