Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change from promptflow-evals to azure-ai-evaluation and add ContentSafetyEvaluator #212

Merged
merged 10 commits into from
Oct 25, 2024

Conversation

slister1001
Copy link
Member

Purpose

Changes dependency from promptflow-evals to azure-ai-evaluation and adds ContentSafetyEvaluator to the ArticleEvaluator.

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

  • Get the code
git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install
  • Test the code
python -m evaluate.evaluate

What to Check

Verify that the following are valid
/eval_results.jsonl

@cassiebreviu cassiebreviu changed the base branch from main to ignite2024 October 22, 2024 14:01
@@ -5,31 +5,31 @@ deployments:
model:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get this to work for me I had to change this file to the following

# yaml-language-server: $schema=ai.yaml.json

deployments:
  - name: gpt-35-turbo
    model:
      format: OpenAI
      name: gpt-35-turbo
      version: "0613"
    sku:
      name: Standard
      capacity: 8
  - name: text-embedding-ada-002
    model:
      format: OpenAI
      name: text-embedding-ada-002
      version: "2"
    sku:
      name: "Standard"
      capacity: 8
  - name: gpt-4
    model:
      format: OpenAI
      name: gpt-4o
      version: "2024-05-13"
    sku:
      name: "GlobalStandard"
      capacity: 8
  - name: gpt-4-evals
    model:
      format: OpenAI
      name: gpt-4
      version: "turbo-2024-04-09"
    sku:
      name: "GlobalStandard"
      capacity: 8

@@ -6,7 +6,7 @@
from pathlib import Path
from datetime import datetime
from promptflow.core import AzureOpenAIModelConfiguration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove this, I don't think AzureOpenAIModelConfiguration is being used.

@marlenezw
Copy link
Contributor

I'm was getting some promptflow rate limit errors. Other than that this works as expected. I will update the PR then merge.

@marlenezw
Copy link
Contributor

marlenezw commented Oct 24, 2024

Worked, but took very long and had lots of rate errors. I'm trying to see why this is happening, but will merge in a bit

Screenshot 2024-10-24 at 11 48 02

@marlenezw
Copy link
Contributor

Merging and will make my recommended changes in the merged branch. Mainly going to stop rate limit errors from showing in the console while I figure out if there's a way to fix them. Thank you for the PR @slister1001.

@marlenezw marlenezw merged commit 537c69e into Azure-Samples:ignite2024 Oct 25, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants