Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Conditions with AND/OR Logic for ISM Transitions #1261

Open
basht0p opened this issue Sep 18, 2024 · 2 comments
Open

Multiple Conditions with AND/OR Logic for ISM Transitions #1261

basht0p opened this issue Sep 18, 2024 · 2 comments
Labels
enhancement New request

Comments

@basht0p
Copy link

basht0p commented Sep 18, 2024

Is your feature request related to a problem?
I mean, to me it's a problem, but no.

What solution would you like?
It would be awesome to be able to apply some AND/OR logic to the conditions for transitions. For example:

"transitions": [
    {
        "state_name": "cold",
        "condition_logic": "and",
        "conditions": {
            "min_index_age": "90d",
            "min_rollover_age": "7d"
        }
    },
    {
        "state_name": "cold",
        "condition_logic": "or",
        "conditions": {
            "min_size": "20gb",
            "min_doc_count": "10000000"
        }
    }
]

In pseudo-query language, this would effectively be the same as:
(min_index_age: 90d AND min_rollover_age: 7d) OR (min_size: 20gb OR min_doc_count: 10000000)

I realize the OR/OR is redundant, as (from what I understand) the ISM already does a sequential "hunt" through the conditions looking for a first match, and I suppose the only novel idea here is adding the AND logic.

My use case, for example, is that I have some index sets which generate up to 40GB of documents per day, and I have others which generate as low as 40GB per quarter. I would like to keep my rotation policy based on size, but I don't want to move an active write index to a cold state, even if it's exceeded 90 days. If it's 101 days old, but is still the active write index, the transition should not apply (hence the additional min_rollover_age).

What alternatives have you considered?
I'm certain the tools exist within OpenSearch to allow me to "tier" my index sets based on their ingress volume/rate/etc, and apply different policies based on that, but that solution feels a bit clunky and I feel as if applying the logic to allow multiple conditions would be a simpler solution overall.

Do you have any additional context?
I am currently on a 5 node cluster running 2.15, however it does not look like this feature exists in later versions.

@vikasvb90
Copy link
Collaborator

vikasvb90 commented Sep 30, 2024

[ Indexing Triage and Backlog Review ]
@basht0p If I understood your use case correctly then I believe you can achieve this already by defining an index management policy which has different states for each transition. For e.g. a state rollover can have just the rollover condition in transition based on index size and second state say cold can have transition condition of min index age. This will behave like an AND operation for the defined index pattern in the policy. For different conditions on a different index pattern, a new policy can be defined.

@basht0p
Copy link
Author

basht0p commented Sep 30, 2024

Honestly, I hadn't thought of that. It's a clever solution that would definitely apply in my use case. That being said, I do still believe that the intuitive expectation when defining multiple conditions is to have some form of logical operator set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New request
Projects
Status: No status
Development

No branches or pull requests

3 participants