Finding an Efficient Way to Ingest S3 Buckets with Datahub

ingestion
1

Original Slack Thread

Hello team, I have a question with S3 ingestion.
I’m trying to add S3 to Datahub, and I want to import each folder to add files in all folder paths in S3
To do that, i have to do like below.

include: 's3://{bucket_name}/folder1/*.*'
include: 's3://{bucket_name}/folder2/*.*'
include: 's3://{bucket_name}/folder1/folder1/*.*'
...```
But i have many buckets to ingestion and it is very complicated
Is ther any other ways?
2

<@U04N9PYJBEW> might help you

Cc: <@UV14447EU>

3

<@U0608DME3UH> have you tried using **?
include: 's3://{bucket_name}/**/*.*'

4

<@U01GZEETMEZ>
I tried, but failed.

Msg is here

attachment
attachment1462×606 72.7 KB

5

and my ingestion source is below

attachment
attachment1380×676 46.4 KB

6

Ah my bad - only exclude paths can use **

You can do something like this

include: 's3://{bucket_name}/*/*.*'
include: 's3://{bucket_name}/*/*/*.*'```
it's not great, but should get the job done
7

Thank you for your help <@U01GZEETMEZ>