Hello Datahub,
I am having an issue with datahub with postgresql, when I deploy datahub helm chart it can’t find table on postgresql database but the postgresql setup job succeeded and I can see it create a table “metadata_aspect_v2” in the database. but datahub upgrade job can’t see it? any clue what’s going on?
javax.persistence.PersistenceException: Query threw SQLException:ERROR: relation "metadata_aspect_v2" does not exist
2024-05-09T14:20:47.020307704+03:00 javax.persistence.PersistenceException: Query threw SQLException:ERROR: relation "metadata_aspect_v2" does not exist
2024-05-09T14:20:47.020316177+03:00 Position: 94 Bind values:[urn:li:dataHubUpgrade:BackfillPolicyFieldsStep, dataHubUpgradeResult, 0] Query was:select urn, aspect, version, metadata, systemMetadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?
2024-05-09T14:20:47.020319768+03:00 at io.ebean.config.dbplatform.SqlCodeTranslator.translate(SqlCodeTranslator.java:80) ~[ebean-api-12.16.1.jar!/:?]
2024-05-09T14:20:47.020323316+03:00 at io.ebean.config.dbplatform.DatabasePlatform.translate(DatabasePlatform.java:245) ~[ebean-api-12.16.1.jar!/:?]
2024-05-09T14:20:47.020327933+03:00 at io.ebeaninternal.server.query.CQueryEngine.translate(CQueryEngine.java:138) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020336076+03:00 at io.ebeaninternal.server.query.DefaultOrmQueryEngine.translate(DefaultOrmQueryEngine.java:43) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020339510+03:00 at io.ebeaninternal.server.core.OrmQueryRequest.translate(OrmQueryRequest.java:58) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020342654+03:00 at io.ebeaninternal.server.query.CQuery.createPersistenceException(CQuery.java:647) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020346075+03:00 at io.ebeaninternal.server.query.CQueryEngine.findMany(CQueryEngine.java:370) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020350879+03:00 at io.ebeaninternal.server.query.DefaultOrmQueryEngine.findMany(DefaultOrmQueryEngine.java:131) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020354008+03:00 at io.ebeaninternal.server.core.OrmQueryRequest.findList(OrmQueryRequest.java:404) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020357164+03:00 at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1463) ~[ebean-core-12.16.1
.jar!/:?]
2024-05-09T14:20:47.020368005+03:00 at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1442) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020371166+03:00 at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1483) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020374323+03:00 at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:447) ~[metadata-io.jar!/:?]
2024-05-09T14:20:47.020377877+03:00 at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:362) ~[metadata-io.jar!/:?]
2024-05-09T14:20:47.020380898+03:00 at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:338) ~[metadata-io.jar!/:?]
2024-05-09T14:20:47.020383899+03:00 at com.linkedin.metadata.entity.EntityServiceImpl.exists(EntityServiceImpl.java:1891) ~[metadata-io.jar!/:?]
2024-05-09T14:20:47.020387626+03:00 at com.linkedin.metadata.entity.EntityService.exists(EntityService.java:60) ~[services.jar!/:?]
2024-05-09T14:20:47.020392276+03:00 at com.linkedin.datahub.upgrade.system.policyfields.BackfillPolicyFieldsStep.skip(BackfillPolicyFieldsStep.java:119) ~[!/:v0.13.2]
2024-05-09T14:20:47.020395367+03:00 at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:63) ~[!/:v0.13.2]
2024-05-09T14:20:47.020398515+03:00 at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:45) ~[!/:v0.13.2]
2024-05-09T14:20:47.020401702+03:00 at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:34) ~[!/:v0.13.2]
2024-05-09T14:20:47.020404754+03:00 at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:86) ~[!/:v0.13.2]
2024-05-09T14:20:47.020407738+03:00 at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:790) ~[spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020411048+03:00 at org.springframework.util.function.ThrowingConsumer$1.acce
ptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.4.jar!/:6.1.4]
2024-05-09T14:20:47.020414253+03:00 at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.4.jar!/:6.1.4]
2024-05-09T14:20:47.020417339+03:00 at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.4.jar!/:6.1.4]
2024-05-09T14:20:47.020420373+03:00 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020423368+03:00 at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020426509+03:00 at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) [spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020429548+03:00 at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
2024-05-09T14:20:47.020432594+03:00 at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[?:?]
2024-05-09T14:20:47.020436594+03:00 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) ~[?:?]
2024-05-09T14:20:47.020443408+03:00 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
2024-05-09T14:20:47.020446698+03:00 at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
2024-05-09T14:20:47.020449782+03:00 at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
2024-05-09T14:20:47.020452766+03:00 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
2024-05-09T14:20:47.020455726+03:00 at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
2024-05-09T14:20:47.020461054+03:00 at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020464215+03:00 at org.springframework
.boot.SpringApplication.run(SpringApplication.java:341) [spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020467298+03:00 at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:149) [spring-boot-3.2.3.jar!/:3.2.3]
2024-05-09T14:20:47.020470275+03:00 at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:42) [!/:v0.13.2]
2024-05-09T14:20:47.020473365+03:00 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2024-05-09T14:20:47.020476478+03:00 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
2024-05-09T14:20:47.020479432+03:00 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
2024-05-09T14:20:47.020483390+03:00 at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
2024-05-09T14:20:47.020486377+03:00 at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91) [datahub-upgrade.jar:v0.13.2]
2024-05-09T14:20:47.020489304+03:00 at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53) [datahub-upgrade.jar:v0.13.2]
2024-05-09T14:20:47.020492324+03:00 at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58) [datahub-upgrade.jar:v0.13.2]
2024-05-09T14:20:47.020495293+03:00 Caused by: org.postgresql.util.PSQLException: ERROR: relation “metadata_aspect_v2” does not exist
2024-05-09T14:20:47.020498502+03:00 Position: 94
2024-05-09T14:20:47.020501794+03:00 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020504804+03:00 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020507788+03:00 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020510894+03:00 at org.postgresql.jdbc.PgStatemen
t.executeInternal(PgStatement.java:490) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020513940+03:00 at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020516900+03:00 at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020520443+03:00 at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119) ~[postgresql-42.3.9.jar!/:42.3.9]
2024-05-09T14:20:47.020523537+03:00 at io.ebean.datasource.pool.ExtendedPreparedStatement.executeQuery(ExtendedPreparedStatement.java:120) ~[ebean-datasource-7.5.jar!/:?]
2024-05-09T14:20:47.020526802+03:00 at io.ebeaninternal.server.query.CQuery.prepareResultSet(CQuery.java:335) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020529800+03:00 at io.ebeaninternal.server.query.CQuery.prepareBindExecuteQueryWithOption(CQuery.java:292) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020532825+03:00 at io.ebeaninternal.server.query.CQuery.prepareBindExecuteQuery(CQuery.java:288) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020535787+03:00 at io.ebeaninternal.server.query.CQueryEngine.findMany(CQueryEngine.java:348) ~[ebean-core-12.16.1.jar!/:?]
2024-05-09T14:20:47.020538885+03:00 … 41 more
Thu, May 9 2024 2:20:47 pm
Hey there! 
Make sure your message includes the following information if relevant, so we can help more effectively!
sql:
datasource:
host: "x.x.x.x:5432"
hostForpostgresqlClient: "x.x.x.x"
port: "5432"
url: "jdbc:postgresql://x.x.x.x:5432/datahub_db"
driver: "org.postgresql.Driver"
username: "user_x"
password:
secretRef: postgresql
secretKey: postgresql
Hello Yassine, did you checked the existence of the table?
Maybe the table was created in another database. I saw that in your configurations you passed url: "jdbc:<postgresql://x.x.x.x:5432/datahub_db>", but by default the setup job creates a database called datahub (<datahub/docker/postgres-setup/env/docker.env at master · datahub-project/datahub · GitHub this env file>). If i’m correct, just changing your database url to “jdbc:postgresql://x.x.x.x:5432/datahub” should work.