[promptflow-evals] evaluator config support #2963
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nagkumar91
previously approved these changes
Apr 23, 2024
promptflow-evals test result 12 files 12 suites 1h 54m 18s ⏱️ Results for commit 5b86460. ♻️ This comment has been updated with latest results. |
nick863
previously approved these changes
Apr 23, 2024
ninghu
changed the title
nick863
previously approved these changes
Apr 25, 2024
singankit
reviewed
Apr 25, 2024
singankit
approved these changes
Apr 26, 2024
nick863
approved these changes
Apr 26, 2024
crazygao
pushed a commit
that referenced
this pull request
May 6, 2024
# Description
Using an example to explain of how evaluator config works in evaluate
API:
Data: col1, col2
Target: col3
Mapping:
question -> ${data.col1}
answer -> ${target.col3}
Evaluate API Workflow
1. Update evaluator config
- Replace all "${target." with "${data."
- New mapping:
question -> ${data.col1}
answer -> ${data.col3}
1. Apply target to data
- New data: col1, col2, col3
1. Column validation
- For each evaluator, rename column:
- col1->question, col3->answer
- question, col2, answer
- Compare with evaluator signature
1. Call evaluator
# All Promptflow Contribution checklist:
- [ ] **The pull request does not introduce [breaking changes].**
- [ ] **CHANGELOG is updated for new features, bug fixes or other
significant changes.**
- [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).**
- [ ] **Create an issue and link to the pull request to get dedicated
review from promptflow team. Learn more: [suggested
workflow](../CONTRIBUTING.md#suggested-workflow).**
## General Guidelines and Best Practices
- [ ] Title of the pull request is clear and informative.
- [ ] There are a small number of commits, each of which have an
informative message. This means that previously merged commits do not
appear in the history of the PR. For more information on cleaning up the
commits in your PR, [see this
page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
### Testing Guidelines
- [ ] Pull request includes test coverage for the included changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Using an example to explain of how evaluator config works in evaluate API:
Data: col1, col2
Target: col3
Mapping:
question -> ${data.col1}
answer -> ${target.col3}
Evaluate API Workflow
Update evaluator config
question -> ${data.col1}
answer -> ${data.col3}
Apply target to data
Column validation
Call evaluator
All Promptflow Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines