MageOS 1.0.4 is not compatible with ES 7, and shoudl drop ES7 modules and support

[ProxiBlue]

Summary

Building a new project, using MageOS 1.0.4, I ran into the problem that certain category views would fail.
This was duplicated in a clean luma 2.4.7 install with sample data, as well as a clean mageOS 1.0.4 install with sample data.

Exception #0 (Elasticsearch\Common\Exceptions\BadRequest400Exception): {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:625] [bool] failed to parse field [must]"}],"type":"x_content_parse_exception","reason":"[1:625] [bool] failed to parse field [filter]","caused_by":{"type":"x_content_parse_exception","reason":"[1:625] [bool] failed to parse field [must]","caused_by":{"type":"illegal_argument_exception","reason":"field name is null or empty"}}},"status":400}

The issue can be tracked down to the fact that 2.4.7 (1.0.4) uses 'must' in the query, and that is an ES 8 convention

So, this is not at all compatible with ES7, and ES 8 is required.

However, once one upgrades ES to v8, a new error is presented:

Catalog Search indexer process unknown error:
{"error":"no handler found for uri [/magento2_product_1_v3/document/_mapping?include_type_name=true] and method [PUT]"}

This can be fixed by setting magento to use 'elasticsearch8' as the needed engine, BUT, that is not possible out-the-box.

Only ES 7 modules are included on base install, and one is required to manually install the ES 8 packages via composer.
This in itself presented a new issue, in the fact that there seems to no mageOS version of the needed package (I know this will be easy to solve, and will need to be solved for the proposed solution)

composer require mage-os/module-elasticsearch-8

results in:

 [InvalidArgumentException]                                                                                                                                                                                      
  Could not find a matching version of package mage-os/module-elasticsearch-8. Check the package spelling, your version constraint and that the package is available in a stability which matches your minimum-s  
  tability (stable).       

so, to get around this for now, I had to add magento repos back into composer

"magento": {
            "type": "composer",
            "url": "https://repo.magento.com/",
            "only": [
                "magento/module-elasticsearch-8"
            ]
        }

however, this then presented a whole new problem:

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Only one of these can be installed: mage-os/module-elasticsearch-7[1.0.4], magento/module-elasticsearch-8[100.4.0, 100.4.1]. They all replace magento/module-elasticsearch-7 and thus cannot coexist.
    - mage-os/product-community-edition 1.0.4 requires mage-os/module-elasticsearch-7 1.0.4 -> satisfiable by mage-os/module-elasticsearch-7[1.0.4].
    - mage-os/product-community-edition is locked to version 1.0.4 and an update of this package was not requested.
    - Root composer.json requires magento/module-elasticsearch-8 ^100.4 -> satisfiable by magento/module-elasticsearch-8[100.4.0, 100.4.1].

which then requires one to replace the magento/module-elasticsearch-7 module in composer

"replace": {
        "mage-os/module-elasticsearch-7": "*"
    },

and then only can one install the needed ES 8 module, and then setup the needed config to use, and overcome the error

This is extremely convoluted to get ES 8 to work.

Examples

See description

Proposed solution

I propose that the module mage-os/module-elasticsearch-7 be removed permanently from mageOS-1.0.4+ and replaced with the v8 version.

End users should not need to go through the whole issue. It should work out-th-box with ES8, as that is the minimum requirement
There should be no need to keep the ES7 module code in the system. It is defunct.

Release note

No response

Triage and priority

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[ProxiBlue]

A further issue has been discovered in teh out-the-box usage of ES 8 with 2.4.7-p2 (unsure if on prior to -p2, have not tested)

All products disappear from display.
Catalog product sql has an errant AND (NULL)

SELECT `e`.*,
       `price_index`.`price`,
       `price_index`.`tax_class_id`,
       `price_index`.`final_price`,
       IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price),
          price_index.min_price)                AS `minimal_price`,
       `price_index`.`min_price`,
       `price_index`.`max_price`,
       `price_index`.`tier_price`,
       IFNULL(review_summary.reviews_count, 0)  AS `reviews_count`,
       IFNULL(review_summary.rating_summary, 0) AS `rating_summary`,
       `stock_status_index`.`stock_status`      AS `is_salable`
FROM `catalog_product_entity` AS `e`
         INNER JOIN `catalog_product_index_price` AS `price_index`
                    ON price_index.entity_id = e.entity_id AND price_index.customer_group_id = 0 AND
                       price_index.website_id = '1'
         LEFT JOIN `review_entity_summary` AS `review_summary`
                   ON e.entity_id = review_summary.entity_pk_value AND review_summary.store_id = 1 AND
                      review_summary.entity_type =
                      (SELECT `review_entity`.`entity_id` FROM `review_entity` WHERE (entity_code = 'product'))
         INNER JOIN `cataloginventory_stock_status` AS `stock_status_index`
                    ON e.entity_id = stock_status_index.product_id
WHERE (stock_status_index.stock_status = 1)
  AND (NULL)

so, to fix, one must adjust ES8 service with a config change

indices:
  id_field_data:
    enabled: true

ref: https://www.sdj.pw/posts/magento2-elasticsearch8/

[rhoerr]

I just ran into the same ES compatibility issue on a site where the host installs ES 8.6 by default. I switched to smile/elasticsuite to handle it, but that's of course not a solution for Mage-OS in general.

[hostep]

so, to fix, one must adjust ES8 service with a config change

indices:
  id_field_data:
    enabled: true

In case this isn't known yet, this has been documented in Magento's docs since December 2022, see: https://experienceleague.adobe.com/en/docs/commerce-operations/upgrade-guide/prepare/prerequisites#upgrade-elasticsearch

[ProxiBlue]

so, to fix, one must adjust ES8 service with a config change

indices:
  id_field_data:
    enabled: true

In case this isn't known yet, this has been documented in Magento's docs since December 2022, see: https://experienceleague.adobe.com/en/docs/commerce-operations/upgrade-guide/prepare/prerequisites#upgrade-elasticsearch

The big diff now looks like ES 8 has it disabled, by default. This is a change people will soon become aware of.