[BUG] Cannot deploy flow to online endpoint (connection refused)

[khangpham-arsen]

Describe the bug
I have followed this https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-deploy-to-code?view=azureml-api-2&tabs=managed to deploy and use PF for quite sometimes. But today I was hit with an error on the Endpoint website stating that "Endpoint unhealthy. Unable to fetch deployment schema."

The logs show that the deployment was successful, but when look at the details, there seems to be a problem:

Container logs:
detecting promptflow run mode...
PROMPTFLOW_RUN_MODE: 
2025-01-15T09:00:45,920649209+00:00 - Starting runsvdir
build info: {"commit": "4f4fc807ae8949466304885a504ec76ca07b76ab", "branch": "n/a", "build_number": "20250109.v2", "date": "2025-01-09 23:10:56"}
start promptflow runtime...
Preparing executor containers...
start embedding store service...
start ingress...
APP_INSIGHTS_KEY is not set!
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow","caller":"compute-runtime-ingress/main.go:49","msg":"Starting server..."}
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow","caller":"compute-runtime-ingress/main.go:58","msg":"App pool config: &{4 /service/app/ promptflow.runtime.app:create_app() root}"}
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow.NewApplicationProcessPool","caller":"gunicornmanager/gunicorn_process_pool.go:123","msg":"Start creating cached gunicorn process"}
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow.NewApplicationProcessPool","caller":"gunicornmanager/gunicorn_process_pool.go:240","msg":"Start creating gunicorn process!","Command":["/azureml-envs/prompt-flow/runtime/bin/gunicorn","-b","127.0.0.1:43573","-t","0","--threads","10","-u","root","--pid","/service/app/flow_pids/43573","promptflow.runtime.app:create_app()"]}
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow.NewApplicationProcessPool","caller":"gunicornmanager/gunicorn_process_pool.go:249","msg":"Gunicorn process with port 43573 started!"}
{"level":"[INFO]","ts":"Jan  15 09:00:45","logger":"PromptFlow","caller":"log/appinsight_metrics.go:38","msg":"Latency between StartIngressServer and StartCreateGunicorn is 0.00 seconds.
"}
{"level":"[WARN]","ts":"Jan  15 09:00:45","logger":"PromptFlow","caller":"util/http_util.go:90","msg":"Error calling endpoint: Get \"http://127.0.0.1:43573/health\": dial tcp 127.0.0.1:43573: connect: connection refused"}
[2025-01-15 09:00:46 +0000] [27] [INFO] Starting gunicorn 22.0.0
[2025-01-15 09:00:46 +0000] [27] [INFO] Listening at: http://127.0.0.1:43573 (27)
[2025-01-15 09:00:46 +0000] [27] [INFO] Using worker: gthread
[2025-01-15 09:00:46 +0000] [28] [INFO] Booting worker with pid: 28
 * Serving Flask app 'promptflow_vectordb.service.server.rest.app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:23333
Press CTRL+C to quit
{"level":"[WARN]","ts":"Jan  15 09:00:51","logger":"PromptFlow","caller":"util/http_util.go:90","msg":"Error calling endpoint: Get \"http://127.0.0.1:43573/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
Failed to prepare executor containers. Exception: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
WARNING:root:'from promptflow import tool' is deprecated and will be removed in the future. Use 'from promptflow.core import tool' instead.
WARNING:root:'from promptflow import ToolProvider' is deprecated and will be removed in the future. Use 'from promptflow.core import ToolProvider' instead.
{"level":"[WARN]","ts":"Jan  15 09:00:56","logger":"PromptFlow","caller":"util/http_util.go:90","msg":"Error calling endpoint: Get \"http://127.0.0.1:43573/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
INFO:CustomEventLogger:StartCreateApp
INFO:CustomEventLogger:FinishCreateApp
2025-01-15 09:00:57 +0000      28 promptflow-system  WARNING  App insights instrumentation key is missing in request header.

How To Reproduce the bug
Follow this https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-deploy-to-code?view=azureml-api-2&tabs=managed.

Screenshots

Test endpoint:

Project structure:

Running Information(please complete the following information):

• Promptflow Package Version using pf -v: 1.16.1
• Operating System: Ubuntu 24.04.1 LTS
• Python Version using python --version: 3.11.11

Additional info

1. flow.flex.yml

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
entry: flow:<>
type: chat
init:
  model_config:
    type: AzureOpenAIModelConfiguration
    default:
      api_key: <>
      azure_endpoint: <>
      api_version: <>
      azure_deployment: <>
  max_total_token:
    type: int
    default: 1024
inputs:
  question:
    type: string
    is_chat_input: true
  chat_history:
    type: list
    default: []
    is_chat_history: true
outputs:
  answer:
    type: string
    is_chat_output: true
environment:
  python_requirements_txt: requirements.txt

2. deployment.yml

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: test-deployment-0115
endpoint_name: test-endpoint
model: azureml:test-model-0115:4
environment:
  build:
    path: image_build_with_requirements
    dockerfile_path: Dockerfile
  inference_config:
    liveness_route:
      path: /health
      port: 8080
    readiness_route:
      path: /health
      port: 8080
    scoring_route:
      path: /score
      port: 8080
instance_type: Standard_E8s_v3
instance_count: 2
request_settings:
  request_timeout_ms: 300000
  max_concurrent_requests_per_instance: 16
app_insights_enabled: true
environment_variables:
  PRT_CONFIG_OVERRIDE: <>
  APPLICATIONINSIGHTS_CONNECTION_STRING: <>
  PF_DISABLE_TRACING: true
  PROMPTFLOW_WORKER_NUM: 8
  PROMPTFLOW_WORKER_THREADS: 16
  PF_LOGGING_LEVEL: ERROR
  PROMPTFLOW_SERVING_ENGINE: fastapi
  APP_INSIGHTS_KEY: <>
properties:
  azureml.promptflow.source_flow_id: test_flow