[SOAR-17874] Mimecast #2897

ablakley-r7 · 2024-10-25T10:47:08Z

Proposed Changes

Description

Describe the proposed changes:

Add 429 error handling task
Now wait 5 minutes before resume
Update Snyk vulnerablity https://www.cve.org/CVERecord?id=CVE-2024-49767

PR Requirements

Developers, verify you have completed the following items by checking them off:

Testing

Unit Tests

Review our documentation on generating and writing plugin unit tests

Unit tests written for any new or updated code

In-Product Tests

If you are an InsightConnect customer or have access to an InsightConnect instance, the following in-product tests should be done:

Screenshot of job output with the plugin changes
Screenshot of the changed connection, actions, or triggers input within the InsightConnect workflow builder

Style

Review the style guide

For dependencies, pin OS package and Python package versions
For security, set least privileged account with USER nobody in the Dockerfile when possible
For size, use the slim SDK images when possible: rapid7/insightconnect-python-3-38-slim-plugin:{sdk-version-num} and rapid7/insightconnect-python-3-38-plugin:{sdk-version-num}
For error handling, use of PluginException and ConnectionTestException
For logging, use self.logger
For docs, use changelog style
For docs, validate markdown with insight-plugin validate which calls icon_validate to lint help.md

Functional Checklist

Work fully completed
Functional
- Any new actions/triggers include JSON test files in the tests/ directory created with insight-plugin samples
- Tests should all pass unless it's a negative test. Negative tests have a naming convention of tests/$action_bad.json
- Unsuccessful tests should fail by raising an exception causing the plugin to die and an object should be returned on successful test
- Add functioning test results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -T tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run -T all --debug --jq (use PR format at end)
- Add functioning run results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -R tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run --debug --jq (use PR format at end)

Assessment

You must validate your work to reviewers:

Run insight-plugin validate and make sure everything passes
Run the assessment tool: insight-plugin run -A. For single action validation: insight-plugin run tests/{file}.json -A
Copy (insight-plugin ... | pbcopy) and paste the output in a new post on this PR
Add required screenshots from the In-Product Tests section

nmccullagh-r7 · 2024-10-25T10:58:33Z

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

+            if rate_limited > now:
+                log_msg += "Still within rate limiting period, skipping task execution..."
+                self.logger.info(log_msg)
+                return [], state, False, 200, None


The only thing about this is that the customer won't have any idea they're rate limited 🤔

unless we put it the task into error state, we currently have no way to display any task info back to idr

If we return a recoverable error message/status code, these should get bubbled in the 'warning' text colour and the integration is still scheduled for the next run

mIght be a PM question to confirm as we haven't done this before, but I agree with Nathanael it would be good to let the customer know they're being rate limited. If we return status code 429, and then the error with cause with something like 'Rate limiting enforced from Mimecast' would show on the UI but let the integration be scheduled on the 5 minute interval, and it could prevent tickets raised for why no data has been pulled?

The only thing is that if we put the task into an error state we won't store the fact that we're rate limited 🤔

I've updated this now. The state should include a new rate limit value.

Based on chat with @nmccullagh-r7 . Update plugin to return 200 on first go, immediately retry and return the new 429. This way the state will be updated to include the retry limit timestamp.

joneill-r7

Current logic looks good, but I'm just wondering if we could improve it a slightly so we base the time on what Mimecast returns and surface this to the customer?

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

joneill-r7 · 2024-10-29T11:56:26Z

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

+            if rate_limited > now:
+                log_msg += "Still within rate limiting period, skipping task execution..."
+                self.logger.info(log_msg)
+                return [], state, False, 200, None


mIght be a PM question to confirm as we haven't done this before, but I agree with Nathanael it would be good to let the customer know they're being rate limited. If we return status code 429, and then the error with cause with something like 'Rate limiting enforced from Mimecast' would show on the UI but let the integration be scheduled on the 5 minute interval, and it could prevent tickets raised for why no data has been pulled?

ablakley-r7 · 2024-11-05T08:46:06Z

Updated error message in PR. Agreed with PM that we can return a 429 for rate limiting errors.

* Add 401 exception handling * Fix it up * Rename response to new response

joneill-r7

LGTM

* Palo alto retry creds 401 (#2826) * Add 401 exception handling * Fix it up * Rename response to new response * Add 429 handler to task | update sdk | add unit test * Update requirements werkzeug * Update rate limit error * Updare bandit * Update error logging * Update value returned by check rate limit status code * Update SDK * Update return 200 first 429 * Update has_more_pages * Update task.py --------- Co-authored-by: Conor <[email protected]>

nmccullagh-r7 reviewed Oct 25, 2024

View reviewed changes

ablakley-r7 force-pushed the soar-17874_mimecast branch from 30b0cc2 to 196d039 Compare October 29, 2024 08:51

ablakley-r7 requested review from nmccullagh-r7, rbowden-r7 and joneill-r7 October 29, 2024 08:52

joneill-r7 reviewed Oct 29, 2024

View reviewed changes

rmurray-r7 force-pushed the develop branch 4 times, most recently from 86876a9 to 9cb0c4a Compare November 4, 2024 15:21

ablakley-r7 force-pushed the soar-17874_mimecast branch from 196d039 to 87a4595 Compare November 4, 2024 15:46

ablakley-r7 requested a review from joneill-r7 November 4, 2024 16:17

igorski-r7 approved these changes Nov 5, 2024

View reviewed changes

rmurray-r7 force-pushed the develop branch 2 times, most recently from e7f10ec to 2e967fc Compare November 5, 2024 09:13

ablakley-r7 force-pushed the soar-17874_mimecast branch from 6899131 to 410384a Compare November 5, 2024 11:30

cmcnally-r7 and others added 8 commits November 5, 2024 11:51

Palo alto retry creds 401 (#2826)

4ab7194

* Add 401 exception handling * Fix it up * Rename response to new response

Add 429 handler to task | update sdk | add unit test

e19900a

Update requirements werkzeug

626f486

Update rate limit error

90a8255

Updare bandit

6e1d1e3

Update error logging

671e332

Update value returned by check rate limit status code

f3107be

Update SDK

cc8f87d

ablakley-r7 force-pushed the soar-17874_mimecast branch from 5869c5d to cc8f87d Compare November 5, 2024 11:51

ablakley-r7 added 2 commits November 5, 2024 13:43

Update return 200 first 429

233fdf9

Update has_more_pages

a96233e

ablakley-r7 force-pushed the soar-17874_mimecast branch from 96bdbd3 to a96233e Compare November 5, 2024 13:56

Update task.py

8c2998d

joneill-r7 approved these changes Nov 5, 2024

View reviewed changes

ablakley-r7 merged commit 0cfafae into develop Nov 5, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOAR-17874] Mimecast #2897

[SOAR-17874] Mimecast #2897

ablakley-r7 commented Oct 25, 2024 •

edited

Loading

nmccullagh-r7 Oct 25, 2024

rbowden-r7 Oct 25, 2024

joneill-r7 Oct 28, 2024

joneill-r7 Oct 29, 2024

nmccullagh-r7 Nov 1, 2024

ablakley-r7 Nov 4, 2024

ablakley-r7 Nov 5, 2024

joneill-r7 left a comment

joneill-r7 Oct 29, 2024

ablakley-r7 commented Nov 5, 2024

joneill-r7 left a comment

[SOAR-17874] Mimecast #2897

[SOAR-17874] Mimecast #2897

Conversation

ablakley-r7 commented Oct 25, 2024 • edited Loading

Proposed Changes

Description

PR Requirements

Testing

Unit Tests

In-Product Tests

Style

Functional Checklist

Assessment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneill-r7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ablakley-r7 commented Nov 5, 2024

joneill-r7 left a comment

Choose a reason for hiding this comment

ablakley-r7 commented Oct 25, 2024 •

edited

Loading