Releases: opensearch-project/opensearch-spark
Releases · opensearch-project/opensearch-spark
Version 0.5.0
What's Changed
- Fix incorrect result in show index statement by @dai-chen in #332
- Bump Flint version to 0.5.0 by @dai-chen in #343
- Enhance index monitor to terminate streaming job on consecutive errors by @dai-chen in #346
- Updating security reachout email by @varun-lodaya in #340
- Transition Flint index state to Failed upon refresh job termination by @dai-chen in #362
- Pre-validate duplicate columns in materialized view query by @dai-chen in #359
- Fix index state stuck in refreshing when streaming job exits early by @dai-chen in #370
- Read dataSourceName from
FlintOptions
and avoid passing as args by @seankao-az in #378 - Refactor static method for OpenSearch client utils by @seankao-az in #377
- [Refactor] Introduce flint-commons for models and interfaces by @noCharger in #373
- Extract metadata log operations from FlintClient into FlintMetadataLogService by @seankao-az in #379
- Support nested indexed field in Flint skipping index SQL statement by @dai-chen in #366
- Support custom metadata log service implementation by @seankao-az in #389
- Abstracting source relations for enhanced covering index rewriting by @dai-chen in #391
- Add OpenSearchCatalog to enable direct access OpenSearch index in Spark SQL by @penghuo in #399
- Enhance Flint Spark API error reporting with centralized handler by @dai-chen in #348
- Unquote text and identifiers in PPL parsing by @seankao-az in #393
- handle MetaException with glue AccessDeniedException by @noCharger in #410
- support shard level split on read path by @penghuo in #402
- Pre-validate checkpoint location write permission by @dai-chen in #414
- Separate metadata log entry data model and persistence by @seankao-az in #406
- Add scheduler_mode index option by @noCharger in #415
- Enhance query rewriter rule to support partial covering index by @dai-chen in #409
- Store error message for streaming job execution in Flint metadata log by @dai-chen in #433
- Add create Pit api and fix sigv4 bug by @penghuo in #434
- Support custom extension conf by @noCharger in #438
- Revert OpenSearch Version to 2.6 by @penghuo in #444
- Remove unimplemented syntax by @ykmr1224 in #439
- Update README to reflect available commands by @ykmr1224 in #447
- Add FlintJob integration test with EMR serverless by @penghuo in #449
- [Bugfix] Insights on query execution error by @noCharger in #475
- Disable unsupported PPL function expressions by @ykmr1224 in #478
- Add error output column to show Flint index statement by @dai-chen in #436
- [Doc] Checklist to fix issue "could not find Docker environment" on macOS by @LantaoJin in #477
- Translate PPL-builtin functions to Spark-builtin functions by @LantaoJin in #448
- Translate Eval Command by @LantaoJin in #499
- Fix SigV4 signature when connecting to OpenSearchServerless by @ykmr1224 in #473
- Support more PPL builtin functions by adding a name mapping by @LantaoJin in #504
- Add OpenSearchTable in flint core by @penghuo in #479
- Reorganize IT directory to prevent unintentional execution from UT by @dai-chen in #501
- Add config on query loop execution frequency by @noCharger in #411
- Use refresh policy from config by @ykmr1224 in #530
- Add PPL describe command by @YANG-DB in #541
- Terminate streaming job when index data is deleted by @dai-chen in #500
- Upgrade Spark 3.5.1 by @penghuo in #525
- [Backport 0.5] Add statement timeout by @opensearch-trigger-bot in #550
- [Backport 0.5] Remove query rewrite for LogsTable skipping index by @opensearch-trigger-bot in #553
- [Backport 0.5] Fix refresh policy back to WAIT_FOR other than writing query result by @opensearch-trigger-bot in #557
- [Backport 0.5] Lateral
eval
expressions are supported after Spark upgrading by @opensearch-trigger-bot in #561 - [Backport 0.5] Translate PPL
dedup
Command Part 1: allowedDuplication=1 by @opensearch-trigger-bot in #566 - [Backport 0.5] Add rate limiter for bulk request by @opensearch-trigger-bot in #571
- [Backport 0.5] Abstract service for accessing Flint index metadata by @opensearch-trigger-bot in #575
- [Backport 0.5] Update PPL describe command by @opensearch-trigger-bot in #578
- [Backport 0.5] Implement stddev_samp and stddev_pop ppl stats function by @opensearch-trigger-bot in #581
- [Backport 0.5]
Top
&Rare
PPL commands support by @opensearch-trigger-bot in #583 - [Backport 0.5] Nested fields query fix by @opensearch-trigger-bot in #585
- [Backport 0.5] Add percentile PPL function by @opensearch-trigger-bot in #587
- [Backport 0.5] Add conf for specifying flint checkpoint location by @opensearch-trigger-bot in #589
- [Backport 0.5] Fix distinct_count ppl stats function by @opensearch-trigger-bot in #590
- [Backport 0.5] PPL Parse command by @opensearch-trigger-bot in #597
- [Backport 0.5] Add support of GroK command including default patterns by @opensearch-trigger-bot in #610
- [Backport 0.5] Translate PPL
dedup
Command Part 2: allowedDuplication>1 by @opensearch-trigger-bot in #615 - [Backport 0.5] Add UT and IT for 2+ level aggregations PPL command by @opensearch-trigger-bot in #613
- [Backport 0.5] Remove checkpoint folder when vacuuming index by @opensearch-trigger-bot in #629
- [Backport 0.5] [Refactor] Extend REPL to support external metadata storage and data storage by @opensearch-trigger-bot in #604
- [Backport 0.5] Update checkpoint location on alter path by @opensearch-trigger-bot in #631
- [Backport 0.5] Refactor FlintJob with FlintStatement and StatementExecutionManager by @opensearch-trigger-bot in #636
- [Backport 0.5] Ppl patterns command by @opensearch-trigger-bot in #639
- [Backport 0.5] Disable timeout params for deleteIndex API only for serverless by @opensearch-trigger-bot in #649
- [Backport 0.5] Add langType to FlintStatement model by @opensearch-trigger-bot in #665
- [Back...
Version 0.4.1
What's Changed
- Bump Flint version to 0.4.1 by @dai-chen in #360
- [Backport 0.4] Transition Flint index state to Failed upon refresh job termination by @opensearch-trigger-bot in #364
- [Backport 0.4] Fix index state stuck in refreshing when streaming job exits early by @opensearch-trigger-bot in #374
- [Backport 0.4] Support nested indexed field in Flint skipping index SQL statement by @opensearch-trigger-bot in #388
- [Backport 0.4] Enhance Flint Spark API error reporting with centralized handler by @dai-chen in #401
- [Backport 0.4] handle MetaException with glue AccessDeniedException by @opensearch-trigger-bot in #412
- [Backport 0.4] [Bugfix] Insights on query execution error by @opensearch-trigger-bot in #486
Full Changelog: v0.4.0...v0.4.1
Version 0.4.0
What's Changed
- Improve pre-validation for Flint index refresh options by @dai-chen in #297
- Remove query log from job executor by @seankao-az in #308
- Adding support to run integ tests on iceberg tables by @asuresh8 in #301
- Bump Flint version to 0.4.0 by @seankao-az in #311
- Allow non-existent checkpoint location path in index validation by @dai-chen in #313
- Clean shuffle data by @penghuo in #312
- Introduce aws sigv4a request signer by @noCharger in #303
- Add covering index based query rewriter rule by @dai-chen in #318
- Add maxExecutors configuration for streaming queries by @penghuo in #326
- add batch_bytes configuration for Flint by @penghuo in #329
- Improve flint error handling by @noCharger in #335
- Apply new logging format to record exceptions by @noCharger in #314
- [Backport 0.4] Enhance index monitor to terminate streaming job on consecutive errors by @opensearch-trigger-bot in #347
Full Changelog: v0.3.0...v0.4.0
Version 0.3.0
What's Changed
- Bump Flint version to 0.3.0 by @penghuo in #258
- Add sql grammar support for show flint index statement by @seankao-az in #266
- Implement BloomFilter query rewrite (without pushdown optimization) by @dai-chen in #248
- Refactor flint log format by @noCharger in #263
- Implement BloomFilter query pushdown optimization by @dai-chen in #271
- Add grammar files for alter index by @seankao-az in #279
- Implement adaptive BloomFilter algorithm by @dai-chen in #251
- Move query from entry point to SparkConf by @noCharger in #274
- Fix spark extension path in README. by @asuresh8 in #282
- Implement show flint index statement by @seankao-az in #276
- Add BloomFilter skipping index SQL support by @dai-chen in #283
- Implement analyze skipping index statement by @rupal-bq in #284
- Reduce default inactivity limit to 3min by @penghuo in #287
- Fix shutdown bug due to non-daemon thread in driver by @kaituo in #292
- Rule out logical deleted skipping index in query rewrite by @dai-chen in #289
- Ignore non-Flint index in show and describe index statement by @dai-chen in #296
- Implement Alter Index SQL statement by @seankao-az in #286
- Welcome new maintainer Louis Chu by @penghuo in #299
- Add AWS credentials provider for metadata access by @noCharger in #285
- Unescape query from EMR spark submit parameter by @seankao-az in #306
- [Backport 0.3] Remove query log from job executor by @opensearch-trigger-bot in #310
- [Backport 0.3] Clean shuffle data by @opensearch-trigger-bot in #322
- [Backport 0.3] Introduce aws sigv4a request signer by @opensearch-trigger-bot in #323
- [Backport 0.3] Add maxExecutors configuration for streaming queries by @opensearch-trigger-bot in #328
- [Manual Backport 0.3] Improve flint error handling (#335) by @noCharger in #338
Full Changelog: v0.2.0...v0.3.0
Version 0.2.0
What's Changed
- Ppl spark join command by @YANG-DB in #69
- Fixed the GitHub id column for Yang-DB in Maintainers by @dtaivpp in #205
- Bump Flint version to 0.2.0 by @dai-chen in #183
- Add vacuum index API and SQL support by @dai-chen in #189
- Restrict the maximum size of value set by default limit by @dai-chen in #208
- bug fix, support array datatype in MV by @penghuo in #211
- Percent-encode invalid flint index characters by @seankao-az in #215
- GHA fix for backport and snapshot-publish by @seankao-az in #222
- Quote table name with backticks to escape special characters in spark.read.table() by @seankao-az in #224
- Configure value set max size in SQL statement by @dai-chen in #210
- Change delete index API to logical delete by @dai-chen in #191
- Trigger barkport workflow when a pull request merges by @noCharger in #230
- Changes for adding default dimensions in CWSink. by @vamsi-amazon in #209
- Refactor Flint index refresh mode by @dai-chen in #228
- Fix Lychee Link Checker Error by @noCharger in #236
- Add OpenSearch metrics by @noCharger in #229
- Fix recover index bug when Flint data index is deleted accidentally by @dai-chen in #241
- Support struct field as indexed column by @dai-chen in #213
- Implement BloomFilter skipping index building logic by @dai-chen in #242
- Support dimension sets in config by @noCharger in #238
- Support on-demand incremental refresh by @dai-chen in #234
- Fix Session state bug and improve Query Efficiency in REPL by @kaituo in #245
- Add interactive job metrics by @noCharger in #240
- Add more flint metrics by @noCharger in #255
Version 0.1.0
Preview release