feat: Enable inference serving capabilities on sagemaker endpoint

[gwang111]

Description of changes:
Added source code to enable serving capabilities on SageMaker Endpoint.

• when the serve command is passed on container startup, the inference server script will execute
• it will then start a Tornado web server in either async or sync mode

Testing
Using a basic LCEL inference script:

logger = logging.getLogger("sagemaker_distribution.inference_server")

qa_prompt = ChatPromptTemplate.from_template("{question}")

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime"
)

text_gen_model = Bedrock(
    model_id="anthropic.claude-v2",
    client=bedrock_runtime
)
text_gen_model.guardrails.update(
    {
        "guardrailVersion": None,
        "guardrailIdentifier": None,
    }
)

lcel_chain = qa_prompt | text_gen_model

def handler(request):
    return lcel_chain.stream(request.body)

• Deployed to SM Endpoint and tested with

invoke_endpoint() -> blocks until all text is received. Still functional however

invoke_endpoint_with_response_stream() -> streaming call works

• Repeated with async iterator and works the same
• Tested with regular/async invoke and all things are functional as well

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[GanAlps]

lets discuss these inputs offline.

[GanAlps]

we will review these with Saurabh/PM since these will be customer facing.

[gwang111]

will schedule a meeting next week

[aws-tianquaw]

Can you move your changes under "template" folder?

• https://github.com/aws/sagemaker-distribution/tree/main/template/v2
• https://github.com/aws/sagemaker-distribution/tree/main/template/v3

For any changes/additions in cpu/gpu.env.in file, please create an issue (example) so that we'll add them to the next image major/minor releases

[gwang111]

Can you move your changes under "template" folder?

• https://github.com/aws/sagemaker-distribution/tree/main/template/v2
• https://github.com/aws/sagemaker-distribution/tree/main/template/v3

For any changes/additions in cpu/gpu.env.in file, please create an issue (example) so that we'll add them to the next image major/minor releases

Moved the code to the v3 template folder since this code will be launching in the next major version release

[cj-zhang]

LGTM! Could you add details about how this was tested though?

[aws-tianquaw]

Please make sure you have E2E integration tests ready for your entrypoint script. And before the automatic test is ready, can you share some manual test results and attach to this PR? For example, you may build an SMD images locally with your PR, and verify using it in sagemaker inference endpoint.

To build images with your changes, you can checkout release-3.0.0 branch and add your changes locally to folder https://github.com/aws/sagemaker-distribution/tree/release-3.0.0/build_artifacts/v3/v3.0/v3.0.0. Then, you can run the following commands to build your image:

conda env update --file environment.yml -n sagemaker-distribution
conda activate sagemaker-distribution
python ./src/main.py build --target-patch-version=3.0.0 --skip-tests