Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write a Github Action that runs the static analysis tests as described in the https://developer.adobe.com/commerce/testing/guide/static/ #4

Open
damienwebdev opened this issue Oct 28, 2022 · 7 comments
Open

Write a Github Action that runs the static analysis tests as described in the https://developer.adobe.com/commerce/testing/guide/static/ #4

damienwebdev opened this issue Oct 28, 2022 · 7 comments
Assignees
@DavidLambauer

Comments

@damienwebdev
Copy link
Member

We should do this for two separate things:

These actions should be added here: https://github.com/graycoreio/github-actions-magento2

  1. The current version in v2.4-develop
  2. The v2.4.5-p1 release

I would expect to see a failure (or a success in Github) that looks like:

Image
@damienwebdev
Copy link
Member Author

Assigning @DavidLambauer

@DavidLambauer
Copy link
Contributor

To be clear, the expected outcome of this issue is to have a GitHub workflow running all the static tests (passing) and publish the build reports. correct?

@damienwebdev
Copy link
Member Author

@DavidLambauer I'm not particularly interested in the build report being publicly accessible in a "pretty format". Whatever the job outputs in CI is enough for me.

@DavidLambauer
Copy link
Contributor

DavidLambauer commented Nov 7, 2022 •
edited
Loading

I started working on this one. Doing some notes-taking here. Don't mind it, yet.

  • I use the graycoreio/github-actions-magento2/supported-version@main to generate the matrix for the currently supported Magento 2 Versions. This action spits out a JSON with all the different PHP/Magento(And most likely more than that) combinations. Then, I can use that output in graycoreio/github-actions-magento2/setup-magento@main to setup a Magento with the parameters that I got from the step before.
  • I wonder if the Matrix is cached somewhere globally, so I can memorize it instead of wasting resources on it?
  • The above setup results in a couple of different jobs. All of them are named similar and a suffix is added. Something like this: setup-magento-extension (magento/project-community-edition:>=2.4.4 <2.4.5). The UI for this is horrible. I wonder if we can simplify the name generation somehow. Something like Magento 2.4.4 / PHP 8.1 would make more sense.

CleanShot 2022-11-07 at 20 13 44

@DavidLambauer
Copy link
Contributor

The env variables for graycoreio/github-actions-magento2/setup-magento@main are a little bit confusing. In particular:

  • There is a mode extension, and there is extensions. extensions refers to the PHP Extensions, so the naming could be php_extensions to make it more clear. The extensions variable does not affect anything right now; it is not used further.
  • The coverage variable expects something like xDebug and not a simple true/false. I had to read the description twice to get it. Either, the description or the naming should be improved here.
  • Some vars are named in snake_case, and some in kebab-case.

@DavidLambauer
Copy link
Contributor

DavidLambauer commented Nov 7, 2022 •
edited
Loading

I have something running as a first draft. As I mentioned, I use the setup-magento actions to get all the different Magento/PHP Combos. Then, I run composer install and run the tests with cd dev/tests/static && php ./../../../vendor/bin/phpunit . testsuite/Magento/Test/Legacy testsuite/Magento/Test/Integrity.

I am not entirely sure what Tests run in Magento. When I look at this I see several static test suites. Digging deeper into the test results from Static tests for CE I can see the Allure Report, but it only mentions Integrity tests.

My latest version: https://github.com/DavidLambauer/mageos-magento2/actions/runs/3414702008/jobs/5682998132

@DavidLambauer
Copy link
Contributor

DavidLambauer commented Nov 8, 2022 •
edited
Loading

Summary of what was done so far:

  • I set up a Workflow that spins up Magento Instances in several different Version Combinations. I use graycoreio/github-actions-magento2/supported-version@main.
  • The Workflow is available here: https://github.com/DavidLambauer/mageos-magento2/actions
  • The Matrix from @damienwebdev spins up 8 different Jobs
  • Each Job spins up a clean Mage-OS
  • Each Job runs the following tests:
    • PHPUnit in dev/tests/unit
    • Integration Integrity Tests from dev/tests/integration/testsuite/Magento/Test/Integrity
    • Legacy Tests in dev/tests/static/testsuite/Magento/Test/Legacy
    • Static Tests in dev/tests/static

I have a couple of open todos and questions:

  • The Matrix compilation should be cached somehow
  • I need some help with the Workflow structure. Right now, everything is just clustered into this single Workflow file. I am sure we need smaller components not to run EVERYTHING ALWAYS.
  • All the tests fail right now. Yet, I don't rely on the phpunit.xml rulesets. I tried reverse engineering the bin/magento dev:tests:run ... command and ended up with an execution that ignores the rulesets.
  • The test output from the Allure Report for Static CE Tests mentions 4018 Tests, whereas my current test suite only contains 3824 tests. Am I missing some?
  • I don't like the current execution. Static tests should be categorized into execution tools. So instead of having one big Static Tests Block, I would go with one for phpmd, phpstan, Copy/Paste Detector, etc...
  • The Allure Reports from Magento right now seem to be broken. They don't list all the test sites but only the integrity tests. This would be nice to have running over all the different test results.
  • The Tests run forever... I can't believe that a single static test run takes more than 3h... Big Yikes...

This was referenced Nov 8, 2022
Closed
Closed
@DavidLambauer DavidLambauer moved this from Todo to In Progress in MageOS - Path to PRs Nov 21, 2022
@DavidLambauer DavidLambauer mentioned this issue Nov 21, 2022
Closed
@damienwebdev damienwebdev mentioned this issue Apr 25, 2023
Open
Vinai pushed a commit that referenced this issue Jan 25, 2024
@vpodorozh @novikor @ihor-sviziev
Integration tests/ fix failed tests (#61)
fb84973 
* Fix \Magento\Paypal\Model\Express\CheckoutTest - dbIsolation missing

* Fix tests Magento\CatalogImportExport\Model\Import\ProductTest\ProductStockTest::testProductStockStatusShouldBeUpdatedOnSchedule
          Zend_Db_Exception: Table "catalog_category_product_index_store0_replica" does not exist

Partially reverting fix from Vinai:
`Create category-product index table for store 0 (#47)

 This fixes a regression introduced by pull request
 https://github.com/mage-os/mageos-magento2/pull/25`

* Fix failing tests with error `undefined key 'product_id`' from app/code/Magento/CatalogInventory/Model/StockManagement.php:110

Test: dev/tests/integration/testsuite/Magento/Downloadable/Block/Sales/Order/Email/Items/Order/DownloadableTest.php

* Fix failed tests: \Magento\Wishlist\Controller\Index\AddTest

Failed due to visibility filter in whishlist collection. Products were not added to root category and not reindexed. So visibility filter ued to exclude them from wishlist items collection: app/code/Magento/Wishlist/Model/ResourceModel/Item/Collection.php:390

* Decrease test-cases - use php 8.2 only for testing purposes
Include and use changed workflow to have more unitary tests executions

* Mark tests as incomplete as actual fix is in: mage-os/mageos-magento-zend-db#1

Error that was fixed:
Unable to revert fixture: Magento/Framework/Backup/_files/trigger.php
#0 /var/www/html/dev/tests/integration/testsuite/Magento/Framework/Backup/DbTest.php(42): Magento\Framework\Backup\DbTest->testBackupAndRollbackIncludesCustomTriggers()
...

* Fix integration tests error:

Magento\Framework\Image\Adapter\InterfaceTest::testRotate with data set #4 ('/var/www/html/dev/tests/integ...st.png', 45, array(157, 35), 'IMAGEMAGICK')
ImagickException: unrecognized color `srgb255,255,255' @ warning/color.c/GetColorCompliance/1064

* Fix integration tests error:

Magento\Framework\Image\Adapter\InterfaceTest::testCreatePngFromString with data set #1 (array(5, 12), array(0, 0, 0), array(0, 20), array(255, 255, 255), 'IMAGEMAGICK')
ImagickException: unable to read font `' @ error/annotate.c/RenderFreetype/1636

* Fix integration tests error:

Magento\Dhl\Model\CarrierTest::testRequestToShip with data set #0 ('GB', 'EU', 'US')
Failed asserting that two DOM documents are equal.

-      <SoftwareName>Magento</SoftwareName>
+      <SoftwareName>Mage-OS</SoftwareName>

* Fix integration tests error:

Magento\Framework\Image\Adapter\InterfaceTest::testCreatePngFromString with data set #1 (array(5, 12), array(0, 0, 0), array(0, 20), array(255, 255, 255), 'IMAGEMAGICK')
PHPUnit\Framework\Exception: Deprecated: Implicit conversion from float 11.34375 to int loses precision in /var/www/html/lib/internal/Magento/Framework/Image/Adapter/ImageMagick.php:505.

* Fix integration tests error:

Magento\Framework\Image\Adapter\InterfaceTest::testCreatePngFromString with data set #1 (array(5, 12), array(0, 0, 0), array(0, 20), array(255, 255, 255), 'IMAGEMAGICK')
PHPUnit\Framework\Exception: Deprecated: Implicit conversion from float 11.34375 to int loses precision in /var/www/html/lib/internal/Magento/Framework/Image/Adapter/ImageMagick.php:505.

* Fix integration tests error:

Magento\Framework\Image\Adapter\InterfaceTest::testRotate with data set #4 ('/var/www/html/dev/tests/integ...st.png', 45, array(157, 35), 'IMAGEMAGICK')
ImagickException: unrecognized color `srgb255,255,255' @ warning/color.c/GetColorCompliance/1064

* Revert "feat(catalog): faster category product count (#25)"

This reverts commit f01535b.

* Revert "Create category-product index table for store 0 (#47)"

This reverts commit 7be2613.

* Add optional param to run tests over specific directory only.

* Fix failed tests: \Magento\Version\Controller\Index\IndexTest

Update package name to mage-os

* Fix integration tests (most likely caused by incorrect merge conflict resolving):

error msg:
1) Magento\Catalog\Block\Adminhtml\Category\Checkboxes\TreeTest::testGetTreeJson
Error: Call to a member function getConnectionName() on null

/var/www/html/lib/internal/Magento/Framework/App/ResourceConnection.php:110
/var/www/html/app/code/Magento/Catalog/Model/ResourceModel/Category/Collection.php:570
...

ref: https://github.com/vpodorozh/mageos-magento2/actions/runs/7472531139/job/20334972566#step:6:39

* Fix failed integration tests in \Magento\Customer

* Revert "Fix failed integration tests in \Magento\Customer"

* Update full-integration-tests.yaml

---------

Co-authored-by: Maksym Novik <[email protected]>
Co-authored-by: Maksym Novik <[email protected]>
Co-authored-by: Ihor Sviziev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DavidLambauer DavidLambauer

Labels
None yet
Projects
MageOS - Path to PRs
Status: In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

resolves #4 : QA Pipeline DavidLambauer/mageos-magento2
2 participants
@DavidLambauer @damienwebdev