Initial commit

LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Peyman Mohseni Kiasari
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Pipfile

@@ -0,0 +1,13 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+pytest = ">=3.6"
+
+[packages]
+gutenberg-cleaner = {editable = true,path = "."}
+
+[requires]
+python_version = "3.6"

README.md

@@ -0,0 +1,43 @@
+![](https://i.ibb.co/sCJXhmz/header-sp.png)
+![](https://img.shields.io/apm/l/vim-mode.svg)
+
+
+# gutenberg-cleaner
+
+a python package for cleaning Gutenberg books and dataset.
+
+### Prerequisites
+nltk package
+
+### Installing
+```
+[sudo] pip install gutenberg-cleaner
+```
+
+## How to use it?
+
+it has two methods called "simple_cleaner" and "super_cleaner".
+### simple_claner:
+Just removes lines that are part of the Project Gutenberg header or footer.
+Doesnt go deeply in the text to remove other things like titles or footnotes or etc...
+```
+simple_cleaner(book: str) -> str
+```
+### super_cleaner:
+Super clean the book (titles, footnotes, images, book information, etc.). may delete some good lines too.
+```
+super_cleaner(book: str, min_token: int = 5, max_token: int = 600) -> str
+```
+min_token: The minimum tokens of a paragraph that is not "dialog" or "quote", -1 means don't tokenize the txt (so it will be faster, but less efficient cleaning).
+max_token: The maximum tokens of a paragraph.
+
+it will mark deleted paragraphs with: [deleted]
+
+
+## Author
+
+* **Peyman Mohseni kiasari**
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details