Super Linter (#21)

* add slim super linter

* correct and ignore linting errors

* update linter settings

* only use black for python linting

.github/ISSUE_TEMPLATE/bug-report.md

@@ -12,6 +12,7 @@ A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:
+
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
@@ -24,15 +25,10 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
+
+- OS: [e.g. iOS]
+- Browser [e.g. chrome, safari]
+- Version [e.g. 22]
 
 **Additional context**
 Add any other context about the problem here.

.github/dependabot.yml

@@ -1,10 +1,11 @@
-version: 2
-updates:
-  - package-ecosystem: "pip" # See documentation for possible values
-    directory: "." # Location of package manifests
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "github-actions" # See documentation for possible values
-    directory: ".github/workflows" # Location of package manifests
-    schedule:
-      interval: "weekly"
+---
+version: 2
+updates:
+  - package-ecosystem: "pip" # See documentation for possible values
+    directory: "." # Location of package manifests
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "github-actions" # See documentation for possible values
+    directory: ".github/workflows" # Location of package manifests
+    schedule:
+      interval: "weekly"

.github/linters/.hadolint.yaml

@@ -0,0 +1,6 @@
+---
+ignored:
+  - DL3006
+  - DL3008
+  - DL3009
+  - DL3059

.github/linters/.markdown-lint.yaml

@@ -0,0 +1,6 @@
+---
+MD013: false
+MD024: false
+MD025: false
+MD034: false
+MD041: false

.github/linters/actionlint.yaml

@@ -0,0 +1 @@
+---

.github/workflows/lint.yaml

@@ -0,0 +1,32 @@
+---
+name: Lint
+
+on:  # yamllint disable-line rule:truthy
+  push: null
+
+jobs:
+  build:
+    name: Lint
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: read
+      packages: read
+      # To report GitHub Actions status checks
+      statuses: write
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Super-linter
+        uses: super-linter/super-linter/[email protected]  # x-release-please-version
+        env:
+          DEFAULT_BRANCH: main
+          # To report GitHub Actions status checks
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_ACTIONS_COMMAND_ARGS: '-ignore SC.*'
+          VALIDATE_PYTHON_FLAKE8: false
+          VALIDATE_PYTHON_ISORT: false
+          VALIDATE_PYTHON_PYLINT: false
+          VALIDATE_PYTHON_MYPY: false

.github/workflows/sync-space.yaml

@@ -1,3 +1,4 @@
+---
 name: Sync Space
 on:
   push:
@@ -13,7 +14,7 @@ jobs:
         fetch-depth: 0
     - name: Login with Huggingface CLI
       run: |
-        pip install -U "huggingface_hub[cli]" hf-transfer
+        pip install -U huggingface_hub[cli] hf-transfer
         git config --global credential.helper store
         huggingface-cli login --token ${{ secrets.HF_TOKEN }}
     - name: Upload Files

.github/workflows/unit-test.yaml

@@ -1,3 +1,4 @@
+---
 name: Unit Tests
 on:
   pull_request:

CODE_OF_CONDUCT.md

@@ -52,7 +52,7 @@ decisions when appropriate.
 
 This Code of Conduct applies within all community spaces, and also applies when
 an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
+Examples of representing our community include using an official email address,
 posting via an official social media account, or acting as an appointed
 representative at an online or offline event.
 
@@ -125,4 +125,4 @@ enforcement ladder](https://github.com/mozilla/diversity).
 
 For answers to common questions about this code of conduct, see the FAQ at
 https://www.contributor-covenant.org/faq. Translations are available at
-https://www.contributor-covenant.org/translations.
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -31,7 +31,7 @@ To ensure a smooth and effective contribution process, please follow these guide
 ### Making Changes
 
 - Fork the repository and create a new branch for your changes.
-- Keep each pull request focused on a single feature or bug fix.
+- Keep each pull request focused on a single feature or bugfix.
 - Write clear and descriptive commit messages.
 - Keep code changes concise and well-documented.
 - Ensure that your code adheres to the project's coding standards.

Dockerfile

@@ -1,6 +1,6 @@
 FROM ollama/ollama
 
-RUN apt-get update -y && apt-get install -y \
+RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
     git \
     python3 \
     python3-pip

README.md

@@ -114,7 +114,7 @@ ollama create volo -f ./Modelfile
 
 1. Download a model of choice from [Huggingface](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) with `git clone https://huggingface.co/<org>/<modelname> model/<modelname>`.
 2. If your model of choice is not in `GGUF` format, convert it with `docker run --rm -v $PWD/model/<modelname>:/model ollama/quantize -q q4_0 /model`.
-2. Modify the [Modelfile's](Modelfile) `FROM` line to contain the path to the `q4_0.bin` file in the modelname directory.
+3. Modify the [Modelfile's](Modelfile) `FROM` line to contain the path to the `q4_0.bin` file in the modelname directory.
 
 ### Create Vector Database
 
@@ -150,7 +150,7 @@ within Kara-Tur.
 
 #### Add Different Data to DB
 
-Choose a new [Filetype Document Loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/) or [App Document Loader](https://python.langchain.com/docs/integrations/document_loaders/) and include those files in your VectorDB.
+Choose a new [File type Document Loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/) or [App Document Loader](https://python.langchain.com/docs/integrations/document_loaders/) and include those files in your VectorDB.
 
 ```python
 Document = namedtuple("Document", ["page_content", "metadata"])

config.yaml

@@ -1,28 +1,29 @@
-data_dir: ./data
-# Huggingface
-embeddings_model: sentence-transformers/all-mpnet-base-v2
-introduction: Ah my good fellow!
-# Sources
-mediawikis:
-  # - dnd4e
-  - dnd5e
-  # - darksun
-  - dragonlance
-  - eberron
-  # - exandria
-  - forgottenrealms
-  - greyhawk
-  # - planescape
-  # - ravenloft
-  # - spelljammer
-# Ollama
-model: volo
-question: How many eyestalks does a Beholder have?
-settings:
-  num_sources: 4
-  repeat_penalty: 1.3
-  temperature: 0.4
-  top_k: 20
-  top_p: 0.35
-# Sources Path
-source: ./sources
+---
+data_dir: ./data
+# Huggingface
+embeddings_model: sentence-transformers/all-mpnet-base-v2
+introduction: Ah my good fellow!
+# Sources
+mediawikis:
+  # - dnd4e
+  - dnd5e
+  # - darksun
+  - dragonlance
+  - eberron
+  # - exandria
+  - forgottenrealms
+  - greyhawk
+  # - planescape
+  # - ravenloft
+  # - spelljammer
+# Ollama
+model: volo
+question: How many eyestalks does a Beholder have?
+settings:
+  num_sources: 4
+  repeat_penalty: 1.3
+  temperature: 0.4
+  top_k: 20
+  top_p: 0.35
+# Sources Path
+source: ./sources

main.py

@@ -1,3 +1,10 @@
+from sys import exit
+
+import argparse
+import chainlit as cl
+import yaml
+import torch
+
 from collections import namedtuple
 from chainlit.input_widget import Slider, TextInput
 from chainlit.playground.config import add_llm_provider
@@ -12,13 +19,7 @@
 from langchain.memory import ChatMessageHistory, ConversationBufferMemory
 from langchain.vectorstores import Chroma
 from provider import LangchainGenericProvider
-from sys import exit
 
-import argparse
-import chainlit as cl
-import yaml
-
-import torch
 
 if not torch.cuda.is_available():
     torch.set_num_threads(torch.get_num_threads() * 2)
@@ -56,7 +57,7 @@ def set_chat_settings(self, settings):
                 setattr(wiki, key, val)
 
 
-### Globals
+# Globals
 wiki = MultiWiki()
 
 
@@ -272,11 +273,10 @@ async def setup_agent(settings):
 
     chain = create_chain()
 
-    if not wiki.question:
-        print("No Prompt for Chatbot found")
+    if wiki.question:
+        res = chain(wiki.question)
+        answer = res["answer"]
+        print(answer)
+        print([source_doc.page_content for source_doc in res["source_documents"]])
+    else:
         exit(1)
-
-    res = chain(wiki.question)
-    answer = res["answer"]
-    print(answer)
-    print([source_doc.page_content for source_doc in res["source_documents"]])

provider.py

@@ -1,9 +1,10 @@
+from typing import List, Union
+
 from chainlit.input_widget import InputWidget
 from chainlit.playground.provider import BaseProvider
 from chainlit.prompt import PromptMessage
 from chainlit.sync import make_async
 from fastapi.responses import StreamingResponse
-from typing import List, Union
 
 
 class LangchainGenericProvider(BaseProvider):
@@ -70,8 +71,7 @@ async def create_completion(self, request):
 
         # https://github.com/langchain-ai/langchain/issues/14980
         result = await make_async(self.llm.stream)(
-            input=messages,
-            **request.prompt.settings
+            input=messages, **request.prompt.settings
         )
 
         def create_event_stream():

test/conftest.py

@@ -1,30 +1,32 @@
 import pytest
 
 optional_markers = {
-    "ollama": {"help": "Test ollama backend with Langchain",
-             "marker-descr": "Enable langchain tests with ollana",
-             "skip-reason": "Test only runs with the --{} option."},
+    "ollama": {
+        "help": "Test ollama backend with Langchain",
+        "marker-descr": "Enable langchain tests with ollana",
+        "skip-reason": "Test only runs with the --{} option.",
+    },
 }
 
 
 def pytest_addoption(parser):
     for marker, info in optional_markers.items():
-        parser.addoption("--{}".format(marker), action="store_true",
-                         default=False, help=info['help'])
+        parser.addoption(
+            "--{}".format(marker), action="store_true", default=False, help=info["help"]
+        )
 
 
 def pytest_configure(config):
     for marker, info in optional_markers.items():
-        config.addinivalue_line("markers",
-                                "{}: {}".format(marker, info['marker-descr']))
+        config.addinivalue_line(
+            "markers", "{}: {}".format(marker, info["marker-descr"])
+        )
 
 
 def pytest_collection_modifyitems(config, items):
     for marker, info in optional_markers.items():
         if not config.getoption("--{}".format(marker)):
-            skip_test = pytest.mark.skip(
-                reason=info['skip-reason'].format(marker)
-            )
+            skip_test = pytest.mark.skip(reason=info["skip-reason"].format(marker))
             for item in items:
                 if marker in item.keywords:
                     item.add_marker(skip_test)

test/test.py

@@ -1,9 +1,8 @@
-from main import MultiWiki, create_chain
-
 import argparse
 import pytest
-
 import torch
+from main import MultiWiki, create_chain
+
 torch.set_num_threads(22)
 
 wiki = MultiWiki()
@@ -36,7 +35,7 @@ def test_multiwiki_set_args():
     parser.add_argument("--no-embed", dest="embed", action="store_false")
     wiki.set_args(parser.parse_args([]))
     print(wiki.args)
-    assert wiki.args.embed == True
+    assert wiki.args.embed is True
 
 
 @pytest.mark.ollama