Write metadata cache data to mappings _meta with refresh time update

[seankao-az]

Description

Metadata Cache Writer

For the most part, same as

• [0.5-nexus] Write mock metadata cache data to mappings _meta #744

In addition to the regular metadata storage using FlintIndexMetadataService, we're dual-writing additional fields, defined by FlintMetadataCache, to the index mappings _meta field. It's intended for frontend users to access some crucial metadata for an index quickly without invoking another backend API call.

This PR adds such fields for all indexes, if the spark config spark.flint.metadataCacheWrite.enabled is set to true.

• _meta.properties.metadataCacheVersion: "1.0"
• _meta.properties.refreshInterval: Integer. Refresh interval of an index measured in seconds. This field is added only if index refresh type is auto refresh and refresh_interval is set
• _meta.properties.sourceTables: Array of Strings. For now, it's mocked data. Update coming in later PR.
• _meta.properties.lastRefreshTime: Long. Timestamp in milliseconds when last refresh happened. This field is added only if index already gets refreshed at least once

Last Refresh Time

Added two new fields in FlintMetadataLogEntry and bumped version of its json doc from 1.0 to 1.1 (because adding new field but not changing existing fields)

• lastRefreshStartTime: Long. Timestamp when last refresh started
• lastRefreshCompleteTime: Long. Timestamp when last refresh completed

These are accurate only for manual refresh (full, incremental) and external scheduler for auto refresh.
For internal scheduler, the jobStartTime (or createTime in FlintMetadataLogEntry) is used to track streaming job start time.

I'm not reusing createTime because they should be updated at different times.
For createTime (for internal scheduler) it's during refreshIndex, recoverIndex, updateIndexManualToAuto
But for lastRefreshStartTime and lastRefreshCompleteTime (for manual refresh and external scheduler) it's only updated in refreshIndex

Related Issues

• [FEATURE] Write metadata to index mappings _meta as read cache for frontend user to access #746

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[seankao-az]

add label to backport to the nexus branch.
To be clear, it shouldn't be backported to 0.5.
The 0.5- part in the name 0.5-nexus is obsolete

[ykmr1224]

Why don't we update for auto refresh?

[seankao-az]

for now only track lastRefreshStartTime and lastRefreshCompleteTime for manual refresh and auto refresh with external scheduler.

for streaming job, we use createTime to track the streaming job start time.
there's no mechanism for tracking start/end time for each micro batch update yet, so updating the 2 timestamp in the refresh could be misleading.

[seankao-az]

I'll add some comment

[seankao-az]

Note to any reviewer if curious, the force push only amended commit 2f58f56 and nothing else

[ykmr1224]

Why do we call it metadata Cache? I was not quite sure the indication of cache.

[seankao-az]

i do welcome a better name... was kind of struggling to come up with a name. I'm not too convinced that MetadataCache is the best one.
So the main use case for this is when using custom index metadata and metadata log storage, these metadata aren't available in OpenSearch index. And some frontend use case need access to these data without making query to backend executed in spark. I interpret it as a read cache for such users, that'll be updated by us dual-writing