OpenAPI Ingest Job Returning Zero Assets despite Successful Job Completion - Forcing Examples Might Help

troubleshoot
1

Original Slack Thread

Hey everyonem does anyoneknow why OpenAPI ingest job might return zero assets? (the api in question is not empty).
When I tried to ingest test api from https://petstore.swagger.io/|https://petstore.swagger.io/ it worked fine and ingested assets, but my ingest job for my company’s api returned empte (ingest job was successful)
here’s my recipe:
source: type: openapi config: name: api_user-summary-statistics_stage url: 'XXX.backend.svc.cluster.local' swagger_file: /swagger.json

Datahub version is 10.5, but I also tried setting CLI to 12.1 and got the same empty result
Full logs are in the thread

2

Hey there! :wave: Make sure your message includes the following information if relevant, so we can help more effectively!

  1. Which DataHub version are you using? (e.g. 0.12.0)
  2. Please post any relevant error logs on the thread!
3

Full ingest logs attachedattachment

4

I think you should set forced_examples for the endpoints:
https://datahubproject.io/docs/generated/ingestion/sources/openapi/#starter-recipe
Ingestion was complaining about this:

              '/api/v1/additional_data/{user_id}': ['No example provided for delete'],
              '/api/v1/get_profiles': ['Unable to find an example for endpoint. Please add it to the list of forced examples.'],
              '/api/v1/privacy_settings': ['Unable to find an example for endpoint. Please add it to the list of forced examples.']},```
5

https://datahubproject.io/docs/generated/ingestion/sources/openapi/#getting-dataset-metadata-from-forced_example

6

<@UV14447EU> it’s still unable to find endpoints, even though I specified the. Didi I do it right? Here’s my recipe, I tried to add several:

    type: openapi
    config:
        name: api_user-summary-statistics_stage
        url: 'XXX'
        swagger_file: /swagger.json
        forced_examples:
            '/api/v1/get_profiles/{status}':
                - active
            '/api/v1/get_profiles{type}':
                - user
            '/api/v1/additional_data/{user_id} ':
                - '0'```
Here's the warnings from logs, full log attached
```      "warnings": {
        "/api/v1/additional_data": [
          "Unable to find an example for endpoint. Please add it to the list of forced examples."
        ],
        "/api/v1/additional_data/{user_id}": [
          "No example provided for delete"
        ],
        "/api/v1/get_profiles": [
          "Unable to find an example for endpoint. Please add it to the list of forced examples."
        ],
        "/api/v1/privacy_settings": [
          "Unable to find an example for endpoint. Please add it to the list of forced examples."
        ]```
![attachment](https://files.slack.com/files-pri/TUMKD5EGJ-F06GBL2S31B/api_logs.log?t=xoxe-973659184562-6705490291811-6708051934148-dd1595bd5f63266bc09e6166373c7a3c)
7

We upgraded to Datahub 12.1 and the result was the same, and I also tried to set CLI to 12.1.4, just in case and still the same result

8

<@UV14447EU> can you take a look at this please?

9

As I see the source try to access your endpoint based on your forced example and it gets 404.
For example based on your url above it tried to access:
your_config_url/api/v1/get_profiles/status/active

10

<@UV14447EU> so does it mean I’m writing examples somehow wrong?
Here’s how it looks in api’s json, if you could check my config in the recipe I’d really appreciate it:pray:
"paths":{"/api/v1/get_profiles":{"get":{"summary":"Get Profiles","operationId":"get_profiles_api_v1_get_profiles_get","parameters":[{"required":true,"schema":{"title":"Ids","type":"array","items":{"maximum":2147483647.0,"exclusiveMinimum":0.0,"type":"integer"}},"name":"ids","in":"query"}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"title":"Response Get Profiles Api V1 Get Profiles Get","type":"array","items":{"$ref":"#/components/schemas/ProfileModel"}}}}},"500":{"description":"Internal Server Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"400":{"description":"Bad Request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}}

11

Can you try to access even from a browser the url:
your_config_url/api/v1/get_profiles/status/active

It seems like our connector tries to access it to get an example but it returns 404

12

Hmm when I try to access it from a browser in gives me “not found” as well… weird, but looks like the problem is on our side. Thanks for your advices though!