Changes in Authentication Requirement for Readiness Endpoint in v0.11.0 Upgrade

Original Slack Thread

Hi!
My team is upgrading to v0.11.0 and we have found that the readiness endpoint (/health/check/ready) now requires authentication. However health endpoint doesn’t.
Is that expected behavior?
Thanks

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Unauthorized to perform this action.</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized to perform this action.</h2>
<table>
<tr><th>URI:</th><td>/health/check/ready</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized to perform this action.</td></tr>
<tr><th>SERVLET:</th><td>openapiServlet</td></tr>
</table>
&lt;hr/&gt;&lt;a href="<https://eclipse.org/jetty>"&gt;Powered by Jetty:// 9.4.46.v20220331&lt;/a&gt;&lt;hr/&gt;

&lt;/body&gt;
&lt;/html&gt;
$
$ curl --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.....FpZJ23BEIILrDOseYGd7qxckaMxe9t1poQHIykq-zYw' <http://demo-datahub-gms:8080/health/check/ready>
{"elasticsearch":{"headers":{},"body":"{\"cluster_name\":\"239618273270:datacatalog-demo-dev\",\"status\":\"yellow\",\"timed_out\":false,\"number_of_nodes\":1,\"number_of_data_nodes\":1,\"discovered_master\":true,\"discovered_cluster_manager\":true,\"active_primary_shards\":159,\"active_shards\":159,\"relocating_shards\":0,\"initializing_shards\":0,\"unassigned_shards\":156,\"delayed_unassigned_shards\":0,\"number_of_pending_tasks\":0,\"number_of_in_flight_fetch\":0,\"task_max_waiting_in_queue_millis\":0,\"active_shards_percent_as_number\":50.476190476190474}","statusCodeValue":200,"statusCode":"OK"}}
$
$ curl -vvv <http://demo-datahub-gms:8080/health>
*   Trying 10.224.214.113:8080...
* Connected to demo-datahub-gms (10.224.214.113) port 8080
&gt; GET /health HTTP/1.1
&gt; Host: demo-datahub-gms:8080
&gt; User-Agent: curl/8.4.0
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 200 OK
&lt; Date: Thu, 19 Oct 2023 14:07:20 GMT
&lt; Content-Length: 0
&lt; Server: Jetty(9.4.46.v20220331)
&lt; 
* Connection #0 to host demo-datahub-gms left intact
demo-datahub-gms-95cf697f8-cskr7:/$ ```

Hey Sergio! <@U03MF8MU5P0> might be able to help you here

I see that it is because the

/health/check/ready
endpoint is part of the OpenAPI stack and not a standalone servlet. The standalone servlet only serves the health check and doesn’t include the AuthorizationFilter. That part is by design so that load balancers, both external and internal to k8, can check the status. I think that the AuthenticationFilter needs some updates to allow specific endpoints unauthenticated since this/health/check/ready endpoint doesn’t seem to return any sensitive data.

> is by design so that load balancers, both external and internal to k8, can check the status
We were using /health for liveness probe and /health/check/ready for rediness probe in k8s.
Although adding the required authentication for the rediness probe is possible, now we are using /health for both, until fixed.

Just created the bug so you can keep track of this https://github.com/datahub-project/datahub/issues/9112

Thanks David and Maggie!

<@U027ZS25RFS> how did you override readiness probe effectively as it’s not supported by values :confused:

i’m facing the same issue!

You can’t overwrite, instead you need to fix in the deployment template here
https://github.com/acryldata/datahub-helm/blob/ee2bc53a7695567072f505f4430829719ee1d6b1/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml#L94|https://github.com/acryldata/datahub-helm/blob/ee2bc53a7695567072f505f4430829719ee[…]/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml

I’ve reverted the template to use the old health check endpoint https://github.com/acryldata/datahub-helm/pull/393