chore: add documentation files and more configuration changes

.pre-commit-config.yaml

@@ -22,15 +22,15 @@ repos:
   hooks:
   - id: prettier
     types_or: [markdown, html, css, scss, javascript, json]
-    args: [--prose-wrap=always]
+    args: [--prose-wrap=preserve]
 
 - repo: https://github.com/Lucas-C/pre-commit-hooks
   rev: v1.1.9
   hooks:
   - id: insert-license
     exclude: ^\..*$
     types: [python]
-    args: [--comment-style, "|#|", --license-filepath, ./LICENSE_HEADER.txt, --fuzzy-match-generates-todo]
+    args: [--comment-style, "|#|", --license-filepath, ./LICENSE_HEADER.txt]
 
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: "v4.6.0"

CHANGELOG.md

@@ -22,4 +22,3 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Removed
 
 - ...
-

CODING_GUIDELINES.md

@@ -0,0 +1,143 @@
+# Coding Guidelines
+
+## Code Style
+
+We follow the [Google Python Style Guide][google-style-guide] with a few minor changes (mentioned below). Since the best way to remember something is to understand the reasons behind it, make sure you go through the style guide at least once, paying special attention to the discussions in the _Pros_, _Cons_, and _Decision_ subsections.
+
+We deviate from the [Google Python Style Guide][google-style-guide] only in the following points:
+
+- We use [`ruff-linter`][ruff-linter] instead of [`pylint`][pylint].
+- We use [`ruff-formatter`][ruff-formatter] for source code and imports formatting, which may work differently than indicated by the guidelines in section [_3. Python Style Rules_](https://google.github.io/styleguide/pyguide.html#3-python-style-rules). For example, maximum line length is set to 100 instead of 79 (although docstring lines should still be limited to 79).
+- According to subsection [_2.19 Power Features_](https://google.github.io/styleguide/pyguide.html#219-power-features), direct use of _power features_ (e.g. custom metaclasses, import hacks, reflection) should be avoided, but standard library classes that internally use these power features are accepted. Following the same spirit, we allow the use of power features in infrastructure code with similar functionality and scope as the Python standard library.
+- According to subsection [_3.19.12 Imports For Typing_](https://google.github.io/styleguide/pyguide.html#31912-imports-for-typing), symbols from `typing` and `collections.abc` modules used in type annotations _"can be imported directly to keep common annotations concise and match standard typing practices"_. Following the same spirit, we allow symbols to be imported directly from third-party or internal modules when they only contain a collection of frequently used typying definitions.
+
+### Common questions
+
+- `pass` vs `...` (`Ellipsis`)
+
+  `pass` is the _no-op_ statement in Python and `...` is a literal value (called _Ellipsis_) introduced for slicing collections of unknown number of dimensions. Although they are very different in nature, both of them are used in places where a statement is required purely for syntactic reasons, and there is not yet a clear standard practice in the community about when to use one or the other. We decided to align with the common pattern of using `...` in the body of empty function definitions working as placeholders for actual implementations defined somewhere else (e.g. type stubs, abstract methods and methods appearing in `Protocol` classes) and `pass` in any other place where its usage is mixed with actual statements.
+
+  ```python
+  # Correct use of `...` as the empty body of an abstract method
+  class AbstractFoo:
+     @abstractmethod
+     def bar(self) -> Bar:
+        ...
+
+  # Correct use of `pass` when mixed with other statements
+  try:
+     resource.load(id=42)
+  except ResourceException:
+     pass
+  ```
+
+### Error messages
+
+Error messages should be written as sentences, starting with a capital letter and ending with a period (avoid exclamation marks). Try to be informative without being verbose. Code objects such as 'ClassNames' and 'function_names' should be enclosed in single quotes, and so should string values used for message interpolation.
+
+Examples:
+
+```python
+raise ValueError(f"Invalid argument 'dimension': should be of type 'Dimension', got '{dimension.type}'.")
+```
+
+Interpolated integer values do not need double quotes, if they are indicating an amount. Example:
+
+```python
+raise ValueError(f"Invalid number of arguments: expected 3 arguments, got {len(args)}.")
+```
+
+The double quotes can also be dropped when presenting a sequence of values. In this case the message should be rephrased so the sequence is separated from the text by a colon ':'.
+
+```python
+raise ValueError(f"unexpected keyword arguments: {', '.join(set(kwarg_names) - set(expected_kwarg_names))}.")
+```
+
+The message should be kept to one sentence if reasonably possible. Ideally the sentence should be kept short and avoid unnecessary words. Examples:
+
+```python
+# too many sentences
+raise ValueError(f"Received an unexpected number of arguments. Should receive 5 arguments, but got {len(args)}. Please provide the correct number of arguments.")
+# better
+raise ValueError(f"Wrong number of arguments: expected 5, got {len(args)}.")
+
+# less extreme
+raise TypeError(f"Wrong argument type. Can only accept 'int's, got '{type(arg)}' instead.")
+# but can still be improved
+raise TypeError(f"Wrong argument type: 'int' expected, got '{type(arg)}'")
+```
+
+The terseness vs. helpfulness tradeoff should be more in favor of terseness for internal error messages and more in favor of helpfulness for `DSLError` and it's subclassses, where additional sentences are encouraged if they point out likely hidden sources of the problem or common fixes.
+
+### Docstrings
+
+TODO: update to autodoc2
+
+We generate the API documentation automatically from the docstrings using [Sphinx][sphinx] and some extensions such as [Sphinx-autodoc][sphinx-autodoc] and [Sphinx-napoleon][sphinx-napoleon]. These follow the Google Python Style Guide docstring conventions to automatically format the generated documentation. A complete overview can be found here: [Example Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html#example-google).
+
+Sphinx supports the [reStructuredText][sphinx-rest] (reST) markup language for defining additional formatting options in the generated documentation, however section [_3.8 Comments and Docstrings_](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) of the Google Python Style Guide does not specify how to use markups in docstrings. As a result, we decided to forbid reST markup in docstrings, except for the following cases:
+
+- Cross-referencing other objects using Sphinx text roles for the [Python domain](https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#the-python-domain) (as explained [here](https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#python-roles)).
+- Very basic formatting markup to improve _readability_ of the generated documentation without obscuring the source docstring (e.g. ` ``literal`` ` strings, bulleted lists).
+
+We highly encourage the [doctest][doctest] format for code examples in docstrings. In fact, doctest runs code examples and makes sure they are in sync with the codebase.
+
+### Module structure
+
+In general, you should structure new Python modules in the following way:
+
+1. _shebang_ line: `#! /usr/bin/env python3` (only for **executable scripts**!).
+2. License header (see `LICENSE_HEADER.txt`).
+3. Module docstring.
+4. Imports, alphabetically ordered within each block (fixed automatically by `ruff-formatter`):
+   1. Block of imports from the standard library.
+   2. Block of imports from general third party libraries using standard shortcuts when customary (e.g. `numpy as np`).
+   3. Block of imports from specific modules of the project.
+5. Definition of exported symbols (optional, mainly for re-exporting symbols from other modules):
+
+```python
+__all__ = ["func_a", "CONST_B"]
+```
+
+6. Public constants and typing definitions.
+7. Module contents organized in a convenient way for understanding how the pieces of code fit together, usually defining functions before classes.
+
+Try to keep sections and items logically ordered, add section separator comments to make section boundaries explicit when needed. If there is not a single evident logical order, pick the order you consider best or use alphabetical order.
+
+Consider configuration files as another type of source code and apply the same criteria, using comments when possible for better readability.
+
+### Ignoring QA errors
+
+You may occasionally need to disable checks from _quality assurance_ (QA) tools (e.g. linters, type checkers, etc.) on specific lines as some tool might not be able to fully understand why a certain piece of code is needed. This is usually done with special comments, e.g. `# noqa: F401`, `# type: ignore`. However, you should **only** ignore QA errors when you fully understand their source and rewriting your code to pass QA checks would make it less readable. Additionally, you should add a short descriptive code if possible (check [ruff rules][ruff-rules] and [mypy error codes][mypy-error-codes] for reference):
+
+```python
+f = lambda: 'empty'  # noqa: E731 [lambda-assignment]
+```
+
+and, if needed, a brief comment for future reference:
+
+```python
+...
+return undeclared_symbol  # noqa: F821 [undefined-name] on purpose to trigger black-magic
+```
+
+## Testing
+
+Testing components is a critical part of a software development project. We follow standard practices in software development and write unit, integration, and regression tests. Note that even though [doctests][doctest] are great for documentation purposes, they lack many features and are difficult to debug. Hence, they should not be used as replacement for proper unit tests except in trivial cases.
+
+<!-- Reference links -->
+
+[doctest]: https://docs.python.org/3/library/doctest.html
+[google-style-guide]: https://google.github.io/styleguide/pyguide.html
+[mypy]: https://mypy.readthedocs.io/
+[mypy-error-codes]: https://mypy.readthedocs.io/en/stable/error_code_list.html
+[pre-commit]: https://pre-commit.com/
+[pylint]: https://pylint.pycqa.org/
+[ruff-formatter]: https://docs.astral.sh/ruff/formatter/
+[ruff-linter]: https://docs.astral.sh/ruff/linter/
+[ruff-rules]: https://docs.astral.sh/ruff/rules/
+[sphinx]: https://www.sphinx-doc.org
+[sphinx-autodoc]: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
+[sphinx-napoleon]: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/index.html#
+[sphinx-rest]: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+[ci-docs]: docs/development/CI/infrastructure.md

CONTRIBUTING.md

@@ -1,20 +1,12 @@
-See the [Scientific Python Developer Guide][spc-dev-intro] for a detailed
-description of best practices for developing scientific packages.
+# Contributing
 
-[spc-dev-intro]: https://learn.scientific-python.org/development/
+JaCe is an open-source project that accepts contributions from any individual or organization. Proper credit will be given to contributors by adding their names to the [AUTHORS.md](AUTHORS.md) file.
 
 # Quick development
 
-The fastest way to start with development is to use nox. If you don't have nox,
-you can use `pipx run nox` to run it without installing, or `pipx install nox`.
-If you don't have pipx (pip for applications), then you can install with
-`pip install pipx` (the only case were installing an application with regular
-pip is reasonable). If you use macOS, then pipx and nox are both in brew, use
-`brew install pipx nox`.
+The fastest way to start with development is to use nox. If you don't have nox, you can use `pipx run nox` to run it without installing, or `pipx install nox`. If you don't have pipx (pip for applications), then you can install with `pip install pipx` (the only case were installing an application with regular pip is reasonable). If you use macOS, then pipx and nox are both in brew, use `brew install pipx nox`.
 
-To use, run `nox`. This will lint and test using every installed version of
-Python on your system, skipping ones that are not installed. You can also run
-specific jobs:
+To use, run `nox`. This will lint and test using every installed version of Python on your system, skipping ones that are not installed. You can also run specific jobs:
 
 ```console
 $ nox -s lint  # Lint only
@@ -23,8 +15,7 @@ $ nox -s docs -- --serve  # Build and serve the docs
 $ nox -s build  # Make an SDist and wheel
 ```
 
-Nox handles everything for you, including setting up an temporary virtual
-environment for each run.
+Nox handles everything for you, including setting up an temporary virtual environment for each run.
 
 # Setting up a development environment manually
 
@@ -34,33 +25,29 @@ You can set up a development environment by running:
 python3 -m venv .venv
 source ./.venv/bin/activate
 pip install --upgrade pip setuptools wheel
-pip install -r requirements-dev.txt 
+pip install -r requirements-dev.txt
 pip install -v -e .
 ```
 
-If you have the
-[Python Launcher for Unix](https://github.com/brettcannon/python-launcher), you
-can instead do:
+If you have the [Python Launcher for Unix](https://github.com/brettcannon/python-launcher), you can instead do:
 
 ```bash
 py -m venv .venv
 py -m pip install --upgrade pip setuptools wheel
-py -m pip install -r requirements-dev.txt 
+py -m pip install -r requirements-dev.txt
 py -m pip install -v -e .
 ```
 
 # Post setup
 
-You should prepare pre-commit, which will help you by checking that commits pass
-required checks:
+You should prepare pre-commit, which will help you by checking that commits pass required checks:
 
 ```bash
 pip install pre-commit # or brew install pre-commit on macOS
 pre-commit install # Will install a pre-commit hook into the git repo
 ```
 
-You can also/alternatively run `pre-commit run` (changes only) or
-`pre-commit run --all-files` to check even without installing the hook.
+You can also/alternatively run `pre-commit run` (changes only) or `pre-commit run --all-files` to check even without installing the hook.
 
 # Testing
 
@@ -94,12 +81,30 @@ nox -s docs -- --serve
 
 # Pre-commit
 
-This project uses pre-commit for all style checking. While you can run it with
-nox, this is such an important tool that it deserves to be installed on its own.
-Install pre-commit and run:
+This project uses pre-commit for all style checking. While you can run it with nox, this is such an important tool that it deserves to be installed on its own. Install pre-commit and run:
 
 ```bash
 pre-commit run -a
 ```
 
 to check all files.
+
+# Pull requests (PRs) and merge guidelines
+
+Before submitting a pull request, check that it meets the following criteria:
+
+1. Pull request with code changes should always include tests.
+2. If the pull request adds functionality, it should be documented both in the code docstrings and in the official documentation.
+3. The pull request should have a proper description of its intent and the main changes in the code. In general this description should be used as commit message if the pull request is approved (check point **5.** below).
+4. If the pull request contains code authored by first-time contributors, they should add their names to the [AUTHORS.md](AUTHORS.md) file.
+5. Pick one reviewer and try to contact them directly to let them know about the pull request. If there is no feedback in 24h/48h try to contact them again or pick another reviewer.
+6. Once the pull request has been approved, it should be squash-merged as soon as possible with a meaningful description of the changes. We use the [Conventional Commits][https://www.conventionalcommits.org/en/v1.0.0/#summary] specification for writing informative and automation-friendly commit messages. The following _commit types_ are accepted:
+   - `chore`: changes that only modify development-related tools, the build system configuration or external dependencies
+   - `ci`: changes to our CI configuration files and scripts
+   - `docs`: documentation only changes
+   - `feat`: a new feature
+   - `fix`: a bug fix
+   - `perf`: a code change that improves performance
+   - `refactor`: a code change that neither fixes a bug nor adds a feature
+   - `style`: changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
+   - `test`: adding missing tests or correcting existing tests

LICENSE_HEADER.txt

@@ -1,6 +1,6 @@
-JaCe - JAX jit using DaCe (Data Centric Parallel Programming)
+JaCe - JAX Just-In-Time compilation using DaCe (Data Centric Parallel Programming)
 
 Copyright (c) 2024, ETH Zurich
 All rights reserved.
 
-SPDX-License-Identifier: BSD-3-Clause
+SPDX-License-Identifier: BSD-3-Clause

README.md

@@ -1,4 +1,12 @@
-# JaCe
+# JaCe - JAX Just-In-Time compilation using DaCe (Data Centric Parallel Programming)
+
+### JAX: High-Performance Array Computing
+
+JAX is a Python library for accelerator-oriented array computation and program transformation, designed for high-performance numerical computing and large-scale machine learning.
+
+### DaCe: Data-Centric Parallel Programming
+
+The DaCe project aims to build new representations for programs and algorithms, in order to efficiently map them to the entire hardware architecture landscape (CPU, GPU, and FPGA) with high utilization. With data-centric parallel programming, we enable direct knowledge transfer of performance optimization, regardless of the scientific application or the target processor.
 
 [![Actions Status][actions-badge]][actions-link]
 [![Documentation Status][rtd-badge]][rtd-link]

docs/conf.py

@@ -1,4 +1,4 @@
-# JaCe - JAX jit using DaCe (Data Centric Parallel Programming)
+# JaCe - JAX Just-In-Time compilation using DaCe (Data Centric Parallel Programming)
 #
 # Copyright (c) 2024, ETH Zurich
 # All rights reserved.
@@ -9,8 +9,9 @@
 
 import importlib.metadata
 
+
 project = "JaCe"
-copyright = "2024, ETH Zurich"
+copyright = "2024, ETH Zurich"  # noqa: A001 [builtin-variable-shadowing]
 author = "ETH Zurich"
 version = release = importlib.metadata.version("jace")
 

noxfile.py

@@ -6,6 +6,7 @@
 
 import nox
 
+
 DIR = Path(__file__).parent.resolve()
 
 nox.needs_version = ">=2024.3.2"
@@ -19,9 +20,7 @@ def lint(session: nox.Session) -> None:
     Run the linter.
     """
     session.install("pre-commit")
-    session.run(
-        "pre-commit", "run", "--all-files", "--show-diff-on-failure", *session.posargs
-    )
+    session.run("pre-commit", "run", "--all-files", "--show-diff-on-failure", *session.posargs)
 
 
 @nox.session
@@ -41,9 +40,7 @@ def docs(session: nox.Session) -> None:
 
     parser = argparse.ArgumentParser()
     parser.add_argument("--serve", action="store_true", help="Serve after building")
-    parser.add_argument(
-        "-b", dest="builder", default="html", help="Build target (default: html)"
-    )
+    parser.add_argument("-b", dest="builder", default="html", help="Build target (default: html)")
     args, posargs = parser.parse_known_args(session.posargs)
 
     if args.builder != "html" and args.serve:
@@ -55,9 +52,7 @@ def docs(session: nox.Session) -> None:
     session.chdir("docs")
 
     if args.builder == "linkcheck":
-        session.run(
-            "sphinx-build", "-b", "linkcheck", ".", "_build/linkcheck", *posargs
-        )
+        session.run("sphinx-build", "-b", "linkcheck", ".", "_build/linkcheck", *posargs)
         return
 
     shared_args = (