Testing strategy for permutations of external git hosts / CI services / CMS versions

[stevector]

Over in pantheon-systems/example-drops-8-composer#61 (comment) @greg-1-anderson highlighted some of the challenges we will have accommodating different services. I agree that tracking all of these permutations will be very complex and difficult.

I think we will benefit from writing out baseline expectations for any given permutation. And I think Behat would be of help here.

Imagine a Behat .feature file meant to lay out the expectations for build-env:create-project. I think it would look something like this:

Feature: create-project
In order to create a new site based on an existing example
As a Pantheon user
I need a command that will automatically connect an external repository, a CI service, Pantheon

Scenario: Basic project
Given a local, authenticated terminus connection
And authentication information for an external git host
And authentication information for an external CI service
And a Pantheon machine token to be used in the to-be-created automation setup

When I run pass the name of an example project to the build-env:create-project command

Then I will have created a new repository on the external git host
And I will have created a new site on Pantheon
And the CMS will have been installed on the Dev environment of the Pantheon Site
And a CI build will have passed on the external CI service.

We could then run this .feature file in a Behat suite per permutation where each suite swapped out different PHP classes that define the appropriate specifics. For instance, we would end up with a Behat context per git host with each context defining the same steps like Given authentication information for an external git host. Of course we would then want this actual Terminus plugin to have a similar set of classes (all implementing the same interface) to define the actual interaction with GitHub/Bitbucket/GitLab.

And yes, running all of the Behat suites would be incredible time consuming. I think we would rely on PRs to this repo running a small subset of permutations and then running the full set on some spaced out schedule.

[stevector]

I should also add that I would expect Behat tests here would be supplemental to unit tests that could run faster and on every PR this this repo.

[greg-1-anderson]

Once I get the basic circle.yml working here to test a single permutation, I think converting it to behat tests as described above would be a good idea. Behat has the nice capability of being able to run through a scenario multiple times, using data from an inline table for each pass through. That could be useful here. I'm not sure if there would be a way to set up a "quick run" that only stepped through some of the available fixture data.

Regarding the support of Bitbucket / Gitlab, my plan was to factor out the code in this plugin into Robo tasks. There could be Github, Bitbucket and Gitlab versions of the same VCS task. (This is just a more specific implementation detail of your suggestion -- Robo tasks are just classes that follow certain conventions.)

As for unit tests, those are best when testing functions with no side effects. There are very few of those here. While mocks have their place (providing values to a function, collecting values from a function), I find very little value in replacing a complex backend with a mock; the result may run very quickly and report very high coverage, but in the end, you have only proved that your implementation was implemented... the way you implemented it. There is no assurance at all that the functional test will still pass, or that the code is still working. Might as well skip those tests.

[stevector]

@greg-1-anderson, I'd like to do an MVP here this week to get #39 merged.

My plan is to use Travis to set up a matrix of options to pass to terminus build:project:create for $SOURCE_COMPOSER_PROJECT

[stevector]

Another option would be to convert the circle.yml file to Circle 2.0 and use Circle 2.0's ability to "fan" out to cover a matrix of options.

[greg-1-anderson]

I agree that we should try to get #39 merged this week. A lot of refactoring has happened in the Build Tools plugin, so we are now quite a bit closer to being able to support multiple repository services and test suites. With the code getting more modular, there might even be some useful unit tests we could write.

[ataylorme]

Cleaning up old issues. There is a CI job per permutation