replace dag flows with flex flows in oob evaluators #3500
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ninghu
reviewed
Jul 3, 2024
...flow-evals/promptflow/evals/evaluators/_content_safety/_content_safety_sub_evaluator_base.py
Outdated
Show resolved
Hide resolved
...flow-evals/promptflow/evals/evaluators/_content_safety/_content_safety_sub_evaluator_base.py
Outdated
Show resolved
Hide resolved
src/promptflow-rag/pyproject.toml
Outdated
| @@ -32,7 +32,7 @@ packages = [ | |||
|
|
|||
| # dependencies | |||
| [tool.poetry.dependencies] | |||
| python = "<4.0,>=3.8" | |||
| python = "<4.0,>=3.8.1" | |||
There was a problem hiding this comment.
I can undo this, since it's not related to the task. But I was noticing version problems when attempting to setup the repo with poetry. This fixed them.
MilesHolland
force-pushed
the
replace-dag-with-flex-in-evals
branch
from
b511a55 to
264db51
Compare
ninghu
reviewed
Jul 8, 2024
| # --------------------------------------------------------- | ||
|
|
||
| # Relative imports don't work for loaded evaluators, so we need absolute imports to be possible. | ||
| from .f1_score import compute_f1_score |
There was a problem hiding this comment.
Can we remove the flow folder? The F1Score implementation is pretty simple, so we can merge all the necessary code into _f1_score.py for easier maintainability.
ninghu
reviewed
Jul 8, 2024
| @@ -9,8 +9,12 @@ | |||
| import requests | |||
| from azure.core.credentials import TokenCredential | |||
| from azure.identity import DefaultAzureCredential | |||
| from constants import EvaluationMetrics, RAIService, Tasks | |||
| from utils import get_harm_severity_level | |||
| try: | |||
There was a problem hiding this comment.
Since we have deprecated dag flow, can we rename the "flow" folder to "common"?
|
replacing with a branch that isn't from a forked repo so things will work. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remove deprecated DAG flags from built-in evaluators in the evals pacakge, replacing them with flex flows. Functionality this amounts to directly coding the behavior specified in former flow.dag.yaml files.
This ended up editing 2 evaluators: F1 score and content safety. In the case of content safety, I used this change to refactor the code slightly into a base class and children, which mostly still exist just to have their own docstrings.