Add python client (#13)

* add python client
Problem: we need to interact with the Flux Restful API from Python
Solution: add a Python client! This includes tests, new documentation,
and (TBA) a release on pypi. I am also adding further spell checking,
tweaking some colors in the docs, and adding additional parameters
to be provide to the flux job submit, including workdir, different node/
gpu / task params, envars, etc..

Signed-off-by: vsoch <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,17 @@
+---
+name: Bug report
+about: Report a bug with flux restful API
+
+---
+
+**Describe the bug**
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Version of action-updater**
+
+Anything else?

.github/ISSUE_TEMPLATE/documentation-or-tutorial-request.md

@@ -0,0 +1,5 @@
+---
+name: Documentation or Tutorial Request
+about: What can we explain better, or what issue did you find?
+
+---

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,13 @@
+---
+name: Feature request
+about: Request a feature
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,5 @@
+---
+name: Question
+about: What's on your mind?
+
+---

.github/workflows/docs.yaml

@@ -42,6 +42,10 @@ jobs:
         cd flux-restful-api
         python3 -m pip install -r requirements.txt
         python3 -m pip install -r .github/dev-requirements.txt
+        # Install flux-restful-client for tutorial rendering
+        cd clients/python/
+        python3 -m pip install .[all]
+        cd -
         cd docs
         python3 -m pip install -r requirements.txt
         make html

.github/workflows/main.yaml

@@ -19,7 +19,7 @@ jobs:
     - name: Check Spelling
       uses: crate-ci/typos@7ad296c72fa8265059cc03d1eda562fbdfcd6df2 # v1.9.0
       with:
-        files: ./README.md
+        files: ./README.md ./docs/
 
     - name: Lint and format Python code
       run: |

.github/workflows/python-tests.yaml

@@ -0,0 +1,44 @@
+name: Test Python flux-restful-api
+
+on:
+  push:
+    branches-ignore:
+    - main
+    - gh-pages
+
+jobs:
+  test-noauth:
+    runs-on: ubuntu-latest
+    services:
+      api:
+        image: ghcr.io/flux-framework/flux-restful-api:latest
+        ports:
+          - 5000:5000
+    steps:
+    - uses: actions/checkout@v3
+    - name: Run tests without auth
+      run: |
+        cd clients/python
+        pip install -e .[all]
+        pytest -xs flux_restful_client/tests/test_api.py
+
+  test-auth:
+    runs-on: ubuntu-latest
+    services:
+      api:
+        env:
+          FLUX_USER: fluxuser
+          FLUX_TOKEN: "12345"
+          FLUX_REQUIRE_AUTH: true
+        image: ghcr.io/flux-framework/flux-restful-api:latest
+        ports:
+          - 5000:5000
+    steps:
+    - uses: actions/checkout@v3
+    - name: Run tests with auth
+      run: |
+        cd clients/python
+        pip install -e .[all]
+        export FLUX_USER=fluxuser
+        export FLUX_TOKEN=12345
+        pytest -xs flux_restful_client/tests/test_api.py

README.md

@@ -15,9 +15,9 @@ With Flux RESTful we can:
 3. option to kill or stop the server (intended for Flux Operator)
 4. allow for start with a user name and token (for basic auth)
 
-
 This project is new and we look forward to [hearing your feedback](https://github.com/flux-framework/flux-restful-api).
-See our ⭐️ [Documentation](https://flux-framework.github.io/flux-restful-api) ⭐️ to get started!
+See our ⭐️ [Documentation](https://flux-framework.github.io/flux-restful-api) ⭐️ to get started,
+and take a look at the [clients](clients) we provide to interact with the server.
 
 ![img/flux-restful.png](img/flux-restful.png)
 

app/routers/api.py

@@ -68,7 +68,14 @@ async def cancel_job(jobid):
     """
     from app.main import app
 
-    flux.job.cancel(app.handle, jobid)
+    try:
+        flux.job.cancel(app.handle, jobid)
+    # This is usually FileNotFoundError
+    except Exception as e:
+        return JSONResponse(
+            content={"Message": "Job cannot be cancelled: %s." % e}, status_code=400
+        )
+
     return JSONResponse(
         content={"Message": "Job is requested to cancel."}, status_code=200
     )
@@ -78,6 +85,10 @@ async def cancel_job(jobid):
 async def submit_job(request: Request):
     """
     Submit a job to our running cluster.
+
+    The logic for parsing a submission is only required here, so we
+    include everything in this function instead of having separate
+    functions.
     """
     from app.main import app
 
@@ -88,19 +99,49 @@ async def submit_job(request: Request):
         )
 
     # Generate the flux job
-    command = shlex.split(payload["command"])
-    fluxjob = JobspecV1.from_command(command=command)
+    command = payload["command"]
+    if isinstance(command, str):
+        command = shlex.split(command)
+
+    # Optional defaults
+    kwargs = {"command": command}
+    for optional in [
+        "num_tasks",
+        "cores_per_task",
+        "gpus_per_task",
+        "num_nodes",
+        "exclusive",
+    ]:
+        if optional in payload and payload[optional]:
+            kwargs[optional] = payload[optional]
+    fluxjob = JobspecV1.from_command(**kwargs)
+
+    # Does the user want a working directory?
+    workdir = payload.get("workdir")
+    if workdir is not None:
+        fluxjob.workdir = workdir
 
     # A duration of zero (the default) means unlimited
     fluxjob.duration = payload.get("runtime", 0) or 0
 
-    # Ensure the cwd is the snakemake working directory
-    # TODO user should be able to provide envars in payload
-    fluxjob.environment = dict(os.environ)
-    flux_future = flux.job.submit_async(app.handle, fluxjob)
+    # Additional envars in the payload?
+    environment = dict(os.environ)
+    extra_envars = payload.get("envars", {})
+    environment.update(extra_envars)
+    fluxjob.environment = environment
+
+    # Submit the job and return the ID, but allow for error
+    try:
+        flux_future = flux.job.submit_async(app.handle, fluxjob)
+    except Exception as e:
+        result = jsonable_encoder(
+            {"Message": "There was an issue submitting that job.", "Error": str(e)}
+        )
+        return JSONResponse(content=result, status_code=400)
+
     jobid = flux_future.get_id()
 
-    # TODO we should write jobid and other metadata to database?
+    # TODO should we write jobid and other metadata to a database?
     result = jsonable_encoder({"Message": "Job submit.", "id": jobid})
     return JSONResponse(content=result, status_code=200)
 

clients/README.md

@@ -0,0 +1,9 @@
+# Flux RESTful API Clients
+
+These are clients to interact with the Flux RESTFul API! We
+keep them versioned alongside the Flux RESTFul API, the reason being
+that we want them to be updated in sync. This also mirrors
+the practice of the parent project [flux-core](https://github.com/flux-framework/flux-core).
+
+ - [Python](python)
+ - Go (coming soon)

clients/python/.gitignore

@@ -0,0 +1,13 @@
+bin/
+lib/
+flux_restful_client.egg-info/
+env
+.env
+build
+release
+dist/
+__pycache__
+*.simg
+*.sif
+*.img
+/.eggs

clients/python/CHANGELOG.md

@@ -0,0 +1,17 @@
+# CHANGELOG
+
+This is a manually generated log to track changes to the repository for each release.
+Each section should include general headers such as **Implemented enhancements**
+and **Merged pull requests**. Critical items to know are:
+
+ - renamed commands
+ - deprecated / removed commands
+ - changed defaults
+ - backward incompatible changes (recipe file format? image file format?)
+ - migration guidance (how to convert images?)
+ - changed behaviour (recipe sections work differently)
+
+The versions coincide with releases on pip. Only major versions will be released as tags on Github.
+
+## [0.0.x](https://github.com/flux-framework/flux-restful-api/tree/main) (0.0.x)
+ - Project skeleton release (0.0.0)