Description

I changed the provision thread queue to 100. This is because currently the size queue of 10 is too low. Basically right now if all the threads are busy and If an 11th request comes, the request will be just lost completely if we have no extra queue or re-route logic of our own. Currently we have no extra re-route logic but I think expanding the queue to a size of 100 to allow more provisioning should be sufficient for now. We should probably come back to this after doing more intense load testing.

I also additionally added a StringToObjectMap parser and builder so we can handle cases where we encounter an Object[] that these objects can be either be Map<String, String> or Map<String, Map<String, String>> like in the case of create connector actions

    "actions": [
        {
            "action_type": "predict",
            "method": "POST",
            "url": "https://${parameters.endpoint}/v1/chat/completions",
            "headers": {
                "Authorization": "Bearer ${credential.openAI_key}"
            },
            "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
        }
    ]

I tested manually through API calls, however I didn't add a UT for the StringToObject because I didn't want to use reflection to check the object type here, since its a generic object.

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.