api endpoints for LLMs

[monperrus]

No description provided.

[monperrus]

Huggingface endpoint

https://api-inference.huggingface.co/models/bigcode/starcoder/

client code
https://huggingface.co/spaces/bigcode/bigcode-playground/blob/2009abb380464f89aba1603069e720f031735cce/app.py#L76

reference: https://huggingface.co/bigcode/starcoder/discussions/30

curl 'https://api-inference.huggingface.co/models/bigcode/starcoder' -X POST   -H 'content-type: application/json' -H "Authorization: Bearer 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" --data-raw '{"inputs":"# python function which adds all elements in a list"}'  

## with full options, returns the logprobs
curl 'https://api-inference.huggingface.co/models/bigcode/starcoder' -X POST   -H 'content-type: application/json' -H "Authorization: Bearer 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" --data-raw '{"inputs":"# python function which adds all elements in a list","parameters": { "best_of": 1, "details": true, "do_sample": true, "max_new_tokens": 2000, "repetition_penalty": 1.03, "return_full_text": true, "seed": null, "stop": [], "temperature": 0.5, "top_k": 10, "top_p": 0.95, "truncate": null, "typical_p": 0.95, "watermark": true }}'

[monperrus]

@khaes-kth the huggingface API returns the logprobs, can be useful for explainability

[khaes-kth]

Interesting! It gives the likelihood per token. Thanks for sharing!

[monperrus]

Code-gpt has encapsulated many Code LLM APIs:
https://github.com/davila7/code-gpt-docs/tree/main/docs/tutorial-ai-providers

[monperrus]

closing, we can put endpoints in #49