[Bug]: Vertex AI tests sometimes don't configure requrements correctly

[tvalentyn]

What happened?

It appears that some Postcommits fail with errors like:

apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
  File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/internal/dill_pickler.py", line 418, in loads
    return dill.loads(s)
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 275, in loads
    return load(file, ignore, **kwds)
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 472, in load
    obj = StockUnpickler.load(self)
  File "/usr/local/lib/python3.8/site-packages/dill/_dill.py", line 462, in find_class
    return StockUnpickler.find_class(self, module, name)
  File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/examples/inference/vertex_ai_image_classification.py", line 35, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

It seems as though we sometimes execute vertex ai tests without providing proper requirements file , or we run vertex ai tests multiple times and nondeterministically choose the test result, and one of them is configured incorrectly.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

Sample failed run:

https://ci-beam.apache.org/job/beam_PostCommit_Python38/4834/testReport/junit/apache_beam.ml.inference.vertex_ai_inference_it_test/VertexAIInference/test_vertex_ai_run_flower_image_classification/

[tvalentyn]

https://console.cloud.google.com/dataflow/jobs/us-central1/2023-11-20_06_37_29-8136332257109490276?project=apache-beam-testing

[tvalentyn]

Successful job:

https://console.cloud.google.com/dataflow/jobs/us-central1/2023-11-19_10_00_34-4583011806972063427

:~$ gsutil cat gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-runner-1119180029-434421-
...
google-cloud-aiplatform>=1.26.0
tensorflow>=2.12.0:~$ 

Failed job has

:~$ gsutil cat gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1120143716-765175-z6xnthva.1700491036.765352/requirements.txt
pyhamcrest!=1.10.0,<2.0.0
mock<3.0.0
parameterized>=0.7.1,<0.8.0

[tvalentyn]

i wonder if there is a race in

beam/sdks/python/scripts/run_integration_test.sh

Line 226 in 32b1067

cp $REQUIREMENTS_FILE postcommit_requirements.txt

that results in some other test suite overriding this file.

[tvalentyn]

Let's reopen if this reappears