Resolve type id 'eclipse-link' as a subtype of MetaStoreManagerFactory

[fivetran-arunsuri]

Description

Resolve type id 'eclipse-link' as a subtype of MetaStoreManagerFactory. Right now with the main branch as it is when we try to configure and run polaris server for production use case with eclipse link as metaStoreManager and postgres as db, it fails with following error

Failed to parse configuration at: metaStoreManager; Could not resolve type id 'eclipse-link' as a subtype of org.apache.polaris.core.persistence.MetaStoreManagerFactory: known type ids = [in-memory] (for POJO property 'metaStoreManager')
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.apache.polaris.service.config.PolarisApplicationConfig["metaStoreManager"])

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• able to start server locally with eclipse link as metastore manager and hit requests to create catalog etc

Checklist:

Please delete options that are not relevant.

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

[MonkeyCanCode]

Hello @fivetran-arunsuri ,

That is expected as Polaris is not enable eclipse-link by default. It is possible to enable it as part of the build. See following for detail (same can be found in https://polaris.apache.org/in-dev/unreleased/metastores/):

Apache Polaris supports the following optional build options:

-PeclipseLink=true – Enables the EclipseLink extension.
-PeclipseLinkDeps=[groupId]:[artifactId]:[version],... – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.

This is archived by the same file you put the PR with following snippet:

if (project.properties.get("eclipseLink") == "true") {
  dependencies { implementation(project(":polaris-eclipselink")) }
}

Reasoning for this is by default eclipselink is configured to use h2 database which is not very useful for prod graded deployment. Due to licensing concern as well as various diff backends end-users may preferred, none of the other database drivers are included as of now. There is a purpose to use quarkus to close this gap (#392), however, that is not yet completed as of today.

Thanks,
Yong Zheng

[fivetran-arunsuri]

Hello @fivetran-arunsuri ,

That is expected as Polaris is not enable eclipse-link by default. It is possible to enable it as part of the build. See following for detail (same can be found in https://polaris.apache.org/in-dev/unreleased/metastores/):

Apache Polaris supports the following optional build options:

-PeclipseLink=true – Enables the EclipseLink extension.
-PeclipseLinkDeps=[groupId]:[artifactId]:[version],... – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.

This is archived by the same file you put the PR with following snippet:

if (project.properties.get("eclipseLink") == "true") {
  dependencies { implementation(project(":polaris-eclipselink")) }
}

Reasoning for this is by default eclipselink is configured to use h2 database which is not very useful for prod graded deployment. Due to licensing concern as well as various diff backends end-users may preferred, none of the other database drivers are included as of now. There is a purpose to use quarkus to close this gap (#392), however, that is not yet completed as of today.

Thanks, Yong Zheng

Hey Yong Zheng,
Thank you for the information. I agree that EclipseLink is currently configured to use the H2 database by default. However, I noticed that the steps to include EclipseLink as a build option are missing from the production documentation here. This also leads to issues because the configuration requires setting up the metastoremanager like this:
metaStoreManager:

type: eclipse-link
 conf-file: META-INF/persistence.xml
 persistence-unit: polaris

in polaris-server.yml, which currently pulls EclipseLink from the build options as you mentioned, creating confusion.

Adding the dependency directly in the PR, as I’ve done, should prevent any errors mentioned in the documentation and keep things straightforward. We could also update the documentation to specify that in case additional backends require dependencies for eg postgres here that can be added to the build file, as our team is currently handling it now. Since we’ll be modifying persistence.xml anyway, this approach seems practical.

I'm unclear on why making EclipseLink optional is necessary. Regardless of the database backend used, this property would still be required for production cases and not the in memory obe, wouldn’t it? Let me know your thoughts.

Thanks,
Arun