Skip to content

Latest commit

 

History

History
131 lines (88 loc) · 9.75 KB

README.md

File metadata and controls

131 lines (88 loc) · 9.75 KB
Raw

Logo

PyPI version PyPI - Python Version

CI Documentation Status codecov License: MIT code style

Many professional data scientists use Jupyter Notebook to accomplish their daily tasks, from preliminary data exploration to model prototyping. Notebooks' interactivity is particularly convenient for data-centric programming and their self-documenting nature provides excellent support for the communication of analytical results.

Nevertheless, Jupyter Notebook has been often criticized for inducing bad programming habits and scarcely supporting Software Engineering best practices. To really benefit from notebooks, users should be aware of their common pitfalls and learn how to prevent them.

In previous work (see "Eliciting Best Practices for Collaboration with Computational Notebooks" [1]), we introduced a catalog of 17 empirically-validated guidelines for the collaborative use of notebooks in a professional context.

To foster the adoption of these best practices, we have created Pynblint, a static analysis tool for Jupyter notebooks written in Python. Pynblint reveals potential notebook defects and recommends corrective actions. It can be operated either as a standalone CLI application or as part of a CI/CD pipeline.

Pynblint screens

The core linting rules of Pynblint have been derived as operationalizations of the best practices from our catalog. Nonetheless, the plug-in architecture of Pynblint enables its users to easily extend the core set of checks with their own linting rules.

Requirements

Python 3.7+.

Installation

Pynblint can be installed with pip or another PyPI package manager:

pip install pynblint

After installation, we recommend exploring the command-line interface of the tool:

pynblint --help

Usage

Pynblint can be used to analyze:

  • a standalone notebook:

    pynblint path/to/the/notebook.ipynb

  • a code repository containing notebooks:

    pynblint path/to/the/project/dir/

    • (possibly also compressed as a .zip archive):

      pynblint path/to/the/compressed/archive.zip

  • a public GitHub repository containing notebooks (support for private repositories is on our roadmap 🙂):

    pynblint --from-github https://github.com/collab-uniba/pynblint

For further information on the available options, please refer to the project documentation.

Catalog of best practices

Pynblint currently implements 22 linting rules:

  • 17 rules applicable to standalone notebooks;
  • 5 rules applicable to entire data science repositories.

These rules are derived from our comprehensive catalog of best practices for collaborative notebook development [1]. Below is the current implementation state of linting rules associated with each best practice:

  • ✅ Complete: Fully implemented
  • 🧩 Partial: Partially implemented with planned improvements
  • ⏳ In Progress: Implementation planned for future releases
  • ❌ Not Planned: No implementation planned
Best Practice from [1] Status Details
1. Use version control ✅ Complete Fully implemented
2. Manage project dependencies ✅ Complete Fully implemented
3. Use self-contained environments ⏳ In Progress: Planned: detection of
- Python virtual environments (e.g., created with venv, pyenv, or conda)
- containerized environments (e.g., Docker).
4. Put imports at the beginning ✅ Complete Fully implemented
5. Ensure re-executability (re-run notebooks top to bottom) ✅ Complete Fully implemented
6. Modularize your code 🧩 Partial Current: detection of modularization constructs in notebooks.
Planned: modularization-focused refactoring recommendations based on detected patterns
7. Test your code 🧩 Partial Current: repository-level detection of coverage data.
Planned: detection of test modules/functions independent of coverage tools
8. Name your notebooks consistently ✅ Complete Fully implemented
9. Stick to coding standards 🧩 Partial Current: detection of cells with invalid Python syntax.
Planned: Integration of Python linters such as pylint, flake8, and ruff
10. Use relative paths ⏳ In Progress: Planned: identification of absolute paths instances in notebooks and recommendation of corresponding relative paths
11. Document your analysis ✅ Complete Fully implemented
12. Leverage Markdown headings to structure your notebook ✅ Complete Fully implemented
13. Keep your notebook clean ✅ Complete Fully implemented
14. Keep your notebook concise ✅ Complete Fully implemented
15. Distinguish production and development artifacts ❌ Not Planned Implementation deemed infeasible (see the note below)
16. Make your notebooks available ⏳ In Progress: Planned: Pynblint will ensure that notebooks marked as published in its configuration are available online
17. Make your data available ✅ Complete Fully implemented

Note: Best practice #15 (Distinguishing Production/Development Artifacts) will not be implemented as there are no objective criteria for determining this distinction through static analysis of notebooks or repositories.

License

This project is licensed under the terms of the MIT license.

References

[1] Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2022. Eliciting Best Practices for Collaboration with Computational Notebooks. Proc. ACM Hum.-Comput. Interact. 6, CSCW1, Article 87 (April 2022), 41 pages. https://doi.org/10.1145/3512934