Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
zouharvi authored Jan 30, 2025
1 parent a431f40 commit 0a4f7c9
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# tokenization-scorer     [![PyPI Version](https://img.shields.io/pypi/v/tokenization-scorer.svg)](https://pypi.python.org/pypi/tokenization-scorer) [![test tokenization-scorer](https://github.com/zouharvi/tokenization-scorer/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/tokenization-scorer/actions/workflows/test.yml)
# tokenization-scorer    

[![PyPI Version](https://img.shields.io/pypi/v/tokenization-scorer.svg)](https://pypi.python.org/pypi/tokenization-scorer)
 
[![test tokenization-scorer](https://github.com/zouharvi/tokenization-scorer/actions/workflows/test.yml/badge.svg)](https://github.com/zouharvi/tokenization-scorer/actions/workflows/test.yml)
 
[![Paper](https://img.shields.io/badge/📜%20paper-481.svg)](https://aclanthology.org/2023.acl-long.284/)

Simple package for evaluating text tokenizations.
The input is a text (list of files or stdin) and output a single number.
Expand Down Expand Up @@ -41,4 +47,4 @@ This package is a side-product of the paper [Tokenization and the Noiseless Chan
year={2023},
url={https://aclanthology.org/2023.acl-long.284/},
}
```
```

0 comments on commit 0a4f7c9

Please sign in to comment.