-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change from promptflow-evals to azure-ai-evaluation and add ContentSafetyEvaluator #212
Conversation
@@ -5,31 +5,31 @@ deployments: | |||
model: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To get this to work for me I had to change this file to the following
# yaml-language-server: $schema=ai.yaml.json
deployments:
- name: gpt-35-turbo
model:
format: OpenAI
name: gpt-35-turbo
version: "0613"
sku:
name: Standard
capacity: 8
- name: text-embedding-ada-002
model:
format: OpenAI
name: text-embedding-ada-002
version: "2"
sku:
name: "Standard"
capacity: 8
- name: gpt-4
model:
format: OpenAI
name: gpt-4o
version: "2024-05-13"
sku:
name: "GlobalStandard"
capacity: 8
- name: gpt-4-evals
model:
format: OpenAI
name: gpt-4
version: "turbo-2024-04-09"
sku:
name: "GlobalStandard"
capacity: 8
@@ -6,7 +6,7 @@ | |||
from pathlib import Path | |||
from datetime import datetime | |||
from promptflow.core import AzureOpenAIModelConfiguration |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should remove this, I don't think AzureOpenAIModelConfiguration
is being used.
I'm was getting some promptflow rate limit errors. Other than that this works as expected. I will update the PR then merge. |
Merging and will make my recommended changes in the merged branch. Mainly going to stop rate limit errors from showing in the console while I figure out if there's a way to fix them. Thank you for the PR @slister1001. |
Purpose
Changes dependency from promptflow-evals to azure-ai-evaluation and adds ContentSafetyEvaluator to the ArticleEvaluator.
Does this introduce a breaking change?
[ ] Yes
[x] No
Pull Request Type
What kind of change does this Pull Request introduce?
How to Test
What to Check
Verify that the following are valid
/eval_results.jsonl