Troubleshooting Azure SQLDB Ingestion with DataHub ODBC 17 Driver

ingestion
1

Original Slack Thread

Hi guys,
datahub 0.10.5.

Question:
Trying to do ingestion of sqldb in Azure using odbc 17 driver and trying to use profiling_pattern:

Issue:
With the profile_pattern set up, nothing is profiled , I can find only that one message in the log
DEBUG {datahub.ingestion.source.sql.sql_common:1038} - Source does not support generating profile candidates.

Any Idea for that?

Config:
source:
type: mssql
config:
host_port: 'IP:PORT'
database: DB_NAME
username: scan-datahub-catalogue
include_views: true
include_tables: true
profiling:
enabled: true
profile_table_level_only: false
profile_pattern:
allow:
- 'dbo.dpe_jda_live.*'
stateful_ingestion:
enabled: false
use_odbc: true
uri_args:
driver: 'ODBC Driver 17 for SQL Server'
autocommit: true
Encrypt: yes
TrustServerCertificate: yes
ssl: true
password: '${MSSQL_PWD}'

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Are you using UI or CLI for ingestion?
  2. Which DataHub version are you using? (e.g. 0.12.0)
  3. What data source(s) are you integrating with DataHub? (e.g. BigQuery)
3

Error: Reference to database and/or server name in 'msdb.dbo.sysjobs
Seems related to fact cross database queries are not allowed in Azure Sql?

4

there is a separate way to connect to MS SQL- try changing your recipe type from mssql to mssql-odbc - what we’ve seen is Azure sources work better with this recipe type