Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adversary Import fails with Code 500 #191

Open
rubentroncon opened this issue Jan 14, 2025 · 3 comments · May be fixed by #192
Open

Adversary Import fails with Code 500 #191

rubentroncon opened this issue Jan 14, 2025 · 3 comments · May be fixed by #192

Comments

@rubentroncon
Copy link

I can import reports and but importing adversaries fails everytime:

[2025-01-14 16:39:17,544] INFO    processor/main       _____ _______  _____   _____   ______ _______
[2025-01-14 16:39:17,544] INFO    processor/main         |   |  |  | |_____] |     | |_____/    |
[2025-01-14 16:39:17,544] INFO    processor/main       __|__ |  |  | |       |_____| |    \_    |
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main         ____  ___    __ __    ___  ____    _____  ____  ____   ____    ___  _____
[2025-01-14 16:39:17,544] INFO    processor/main        /    T|   \  |  T  |  /  _]|    \  / ___/ /    T|    \ l    j  /  _]/ ___/
[2025-01-14 16:39:17,544] INFO    processor/main       Y  o  ||    \ |  |  | /  [_ |  D  )(   \_ Y  o  ||  D  ) |  T  /  [_(   \_
[2025-01-14 16:39:17,544] INFO    processor/main       |     ||  D  Y|  |  |Y    _]|    /  \__  T|     ||    /  |  | Y    _]\__  T
[2025-01-14 16:39:17,544] INFO    processor/main       |  _  ||     |l  :  !|   [_ |    \  /  \ ||  _  ||    \  |  | |   [_ /  \ |
[2025-01-14 16:39:17,544] INFO    processor/main       |  |  ||     | \   / |     T|  .  Y \    ||  |  ||  .  Y j  l |     T\    |
[2025-01-14 16:39:17,544] INFO    processor/main       l__j__jl_____j  \_/  l_____jl__j\_j  \___jl__j__jl__j\_j|____jl_____j \___j
[2025-01-14 16:39:17,544] INFO    processor/main       
[2025-01-14 16:39:17,544] INFO    processor/main       Start Threat Actor galaxy cluster alignment
[2025-01-14 16:39:17,545] INFO    processor/main       Retrieving all adversaries.
[2025-01-14 16:39:18,057] INFO    processor/main       Got 257 adversaries from the Crowdstrike Intel API.
[2025-01-14 16:39:18,451] INFO    processor/main       Retrieving all adversaries.
Traceback (most recent call last):
  File "MISP-tools/misp_import.py", line 505, in <module>
    main()
  File "/MISP-tools/misp_import.py", line 497, in main
    import_handler.build()
  File "/MISP-tools/misp_import.py", line 401, in build
    self.import_new_events()
  File "/MISP-tools/misp_import.py", line 387, in import_new_events
    self.importer.import_from_crowdstrike(
  File "/MISP-tools/cs_misp_import/importer.py", line 342, in import_from_crowdstrike
    self.actors_importer.process_actors(actors_days_before, self.event_ids)
  File "/MISP-tools/cs_misp_import/actors.py", line 184, in process_actors
    cluster_result = self.misp.add_galaxy_cluster(get_threat_actor_galaxy_id(self.misp), cluster)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/misp-modules/lib/python3.11/site-packages/pymisp/api.py", line 1825, in add_galaxy_cluster
    cluster_j = self._check_json_response(r)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/misp-modules/lib/python3.11/site-packages/pymisp/api.py", line 3978, in _check_json_response
    r = self._check_response(response, expect_json=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/MISP-tools/cs_misp_import/misp_safe_check_response.py", line 55, in safe_check_response
    raise MISPServerError(fail_msg)
pymisp.exceptions.MISPServerError: Error code 500: SQLSTATE[01000]: Warning: 1265 Data truncated for column &#039;galaxy_id&#039; at row 1

In the MISP error.log I see:

2025-01-14 15:34:03 Error: [PDOException] SQLSTATE[01000]: Warning: 1265 Data truncated for column 'galaxy_id' at row 1
Request URL: /galaxy_clusters/add/698774c7-8022-42c4-917f-8d6e4f06ada3
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Datasource/Database/MysqlObserverExtended.php(162): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1132): MysqlObserverExtended->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1943): DboSource->create()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1761): Model->_doSave()
#5 /var/www/MISP/app/Model/GalaxyCluster.php(342): Model->save()
#6 /var/www/MISP/app/Controller/GalaxyClustersController.php(321): GalaxyCluster->saveCluster()
#7 [internal function]: GalaxyClustersController->add()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(500): ReflectionMethod->invokeArgs()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#11 /var/www/MISP/app/webroot/index.php(107): Dispatcher->dispatch()
#12 {main}
@DeZuko
Copy link

DeZuko commented Jan 15, 2025 •
edited
Loading

I also got similar kind of error but when importing indicators:

[2025-01-15 15:21:26,298] INFO     config  _______ _     _ _______ _______ _     _      _______  _____  __   _ _______ _____  ______
[2025-01-15 15:21:26,298] INFO     config  |       |_____| |______ |       |____/       |       |     | | \  | |______   |   |  ____
[2025-01-15 15:21:26,298] INFO     config  |_____  |     | |______ |_____  |    \_      |_____  |_____| |  \_| |       __|__ |_____|
[2025-01-15 15:21:26,298] INFO     config
[2025-01-15 15:21:26,299] WARNING  config  misp_enable_ssl                             SSL is disabled for MISP API requests
[2025-01-15 15:21:27,024] INFO     config  No configuration errors found (1 warning)
[2025-01-15 15:21:27,024] INFO     config
[2025-01-15 15:21:27,024] INFO     config  ____ _  _ ____ ____ _  _ ____    ___  ____ ____ ____ ____ ___
[2025-01-15 15:21:27,024] INFO     config  |    |__| |___ |    |_/  [__     |__] |__| [__  [__  |___ |  \
[2025-01-15 15:21:27,024] INFO     config  |___ |  | |___ |___ | \_ ___]    |    |  | ___] ___] |___ |__/
[2025-01-15 15:21:27,024] INFO     config
/home/misp/csenv/lib/python3.10/site-packages/pymisp/__init__.py:67: FutureWarning: This class is deprecated, use PyMISP instead
  warnings.warn('This class is deprecated, use PyMISP instead', FutureWarning)
[2025-01-15 15:21:28,066] INFO    processor/main       Retrieving all galaxy cluster values for the Android cluster.
Traceback (most recent call last):
  File "/home/misp/MISP-tools/misp_import.py", line 505, in <module>
    main()
  File "/home/misp/MISP-tools/misp_import.py", line 495, in main
    import_handler = ImportHandler(config, intel_api_client,
  File "/home/misp/MISP-tools/misp_import.py", line 312, in __init__
    self.importer = CrowdstrikeToMISPImporter(
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 76, in __init__
    self.all_galaxies = self.get_galaxies()
  File "/home/misp/MISP-tools/cs_misp_import/importer.py", line 270, in get_galaxies
    all_galaxies.append(self.misp_client.search_galaxy_clusters(gal["id"], searchall=""))
  File "/home/misp/csenv/lib/python3.10/site-packages/pymisp/api.py", line 1781, in search_galaxy_clusters
    clusters_j = self._check_json_response(r)
  File "/home/misp/csenv/lib/python3.10/site-packages/pymisp/api.py", line 3978, in _check_json_response
    r = self._check_response(response, expect_json=True)
  File "/home/misp/MISP-tools/cs_misp_import/misp_safe_check_response.py", line 55, in safe_check_response
    raise MISPServerError(fail_msg)
pymisp.exceptions.MISPServerError: Error code 500: An Internal Error Has Occurred.

My MISP error.log shows:

2025-01-15 15:21:28 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'Galaxy.default' in 'where clause'
Request URL: /galaxy_clusters/index/3
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1226): DboSource->fetchAll()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3053): DboSource->read()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3025): Model->_readDataSource()
#6 /var/www/MISP/app/Model/AppModel.php(4359): Model->find()
#7 /var/www/MISP/app/Controller/GalaxyClustersController.php(105): AppModel->find()
#8 [internal function]: GalaxyClustersController->index()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs()
#10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#11 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#12 /var/www/MISP/app/webroot/index.php(105): Dispatcher->dispatch()
#13 {main}

I realize this happen after I upgraded my MISP version to 2.4.200

@rubentroncon
Copy link
Author

I also got similar kind of error but when importing indicators:

I realize this happen after I upgraded my MISP version to 2.4.200

Actually, your specific issue is also reported in the MISP repository: MISP/MISP#10062
I don't know how hard it is to patch the SQL query yourself, but good luck :-)

rubentroncon added a commit to rubentroncon/MISP-tools that referenced this issue Jan 16, 2025
@rubentroncon
Update helper.py
5029ce2 
Edited Line 169:
Fixed [Issue 191](CrowdStrike#191)
Changed `ta_galaxy_id = gal["Galaxy"]["uuid"]` to `ta_galaxy_id = gal["Galaxy"]["id"]`
@rubentroncon rubentroncon linked a pull request Jan 16, 2025 that will close this issue
Open
@rubentroncon
Copy link
Author

Fixed my issue by editing line 169 in cs_misp_import/helper.py and setting ta_galaxy_id = gal["Galaxy"]["uuid"] to ta_galaxy_id = gal["Galaxy"]["id"]

The error suggested that the Galaxy ID used to create the GalaxyCluster was too long.

Taking a look at the galaxy_clusters table, a Galaxy ID was max. 11 integers, but when debugging I saw a UUID being used to add the GalaxyCluster.

Created Pull request 192.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update helper.py rubentroncon/MISP-tools
2 participants
@rubentroncon @DeZuko