Set up a process to roll Gallery version in Flutter Devicelab tests

[pennzht]

Is your feature request related to a problem? Please describe.
(Please add)

Describe the solution you'd like
It would be great to have a process to roll the Gallery version regularly in Devicelab Tests in Flutter.

Describe alternatives you've considered
(Please add)

Additional context
Please see flutter/flutter#62986 for more discussion.

[jmagman]

Copying discussion from flutter/flutter#62986

Is there any way we can move these tests to customer tests https://github.com/flutter/tests/blob/master/registry/flutter_gallery.test and out of Flutter proper? That is also pinning a SHA that already needs to roll.

https://github.com/flutter/tests#flutter-tests

This repository contains references to tests that are run with every commit to Flutter to verify that no breaking changes have been introduced. The tests referenced by this repository are typically maintained by people outside of the Flutter team, as part of the development of their applications. They are intended to give the Flutter team visibility into how their changes affect real-world developers using Flutter.

flutter_gallery_v2_chrome_run_test particularly looks look a good candidate. flutter_gallery_v2_web_compile_test is gathering benchmarks for compilation times, and I don't think there's a concept of benchmark tracking in customer_testing. However we should weigh the value of tracking compilation times for the new gallery (versus any of the other integration tests we already have in the flutter repo and not in a different repo) versus the complexity of maintaining this new "roll".

\cc @johnsonmh @chunhtai @Hixie

[jmagman]

There's also duplicate flutter/flutter#63004.

[christopherfujino]

i'll close mine, thanks for this @pennzht

[xster]

Echoing @jmagman's suggestion.

The other ramification is that having a benchmark test driving the gallery in Flutter repo (probably unintentionally) actually makes Flutter Gallery the one true source of all dependency and transitive dependencies of all flutter/flutter repo. Since flutter/flutter enforces a single resolved version for all dependencies, and since the web macrobenchmark depends on a Flutter Gallery's git hash commit, that Flutter Gallery's pubspec.yaml effectively becomes the locked dependency version that all dependencies in flutter/flutter have to use.

This 1) creates resolution issues that might arise when changing pubspec in Flutter Gallery that will have bigger downstream ramifications when rolling and 2) changes the order of package upgrades in flutter/flutter. The sequencing will have to then be send a PR to flutter/gallery to update pubspec in order to be able to change the dependency in flutter/flutter.

[Hixie]

I think we should seriously consider not testing or benchmarking the "real" gallery as part of Flutter's regular integration tests. Things we want to have be part of our integration tests should be in the main repo. If we want to make sure we don't break it, we could add it to the customer_testing shard (flutter/tests).

[jmagman]

@Hixie Parts of it already are in the customer_testing shard. Check out my comment above about what should be easy to move and what would be trickier.

[Hixie]

Excellent. I think it's fine for us to remove the other testing we have right now. The cost is disproportionate to the benefit and we are so deep in debt here that the benefit of reducing the cost would offset even a non-trivial benefit.

[yjbanov]

@Hixie what is it about the customer_testing shard that would make it easier to maintain? I'm curious if we could apply the same method in the benchmark. This way we would still be able to benchmark the gallery without the drawbacks of whatever it is we're doing currently.