Issues with SQL Database Scanning in DataHub Version 0.10.3

troubleshoot
1

Original Slack Thread

Hi all,
running, DH version 0.10.3.
and having issues while scanning SQL Database.
Sometimes, during ingest process, especially while scanning larger data sources, we are facing following issue for multiple datasets:

                        'metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and '
                        'version=?',
               'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
                        'message': 'javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, '
                                   'createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?',
                        'status': 500,
                        'id': 'urn:li:dataset:(urn:li:dataPlatform:mssql,TestingResource'
                              'WMS.dscusnoramdatamartwmslssqldb01.DLXUSJNJP2.PRTMST_WH,PROD)'}},

 Pipeline finished with at least 793 failures; produced 25301 events in 1 hour, 15 minutes and 21.68 seconds.```
 In Catalog, the mentioned entity exists, but there is no attribute schema assigned, looks like empty.

 When Im trying to rescan the particular schema only (by filtering in ingest configuration)  - DLXUSJNJP2, its scan completely ok with no error and in data catalog all looks ok.
 It happens randomly.
2

<@U01GZEETMEZ> might be able to speak to this!

3

We improved the backend stability in 0.10.5, so you should stop seeing this issue once you upgrade!

4

Hello <@U01GZEETMEZ>,
As <Slack said> in <#CUMUWQU66|ingestion> channel, this issue unfortunately still occurs with v0.10.5 version of DataHub.
And this, with any kind of ingestion source, which makes it more and more critical, don’t you think?