Fix: Database Schema #9

shiv810 · 2024-09-09T05:25:28Z

Resolves #8

Updated migrations for the database
Supports NULL comment body for private repository.
Has timestamps now for creation and modification.
PKey is Github's global node id. No need for Organization or Repo Name for referencing it.
Schema has author_id now.

src/handlers/add-comments.ts

src/types/database.ts

tests/__mocks__/strings.ts

src/adapters/supabase/helpers/comment.ts

0x4007 · 2024-09-09T06:04:19Z

Why did you switch to main from development that doesn't seem right

shiv810 · 2024-09-09T07:22:04Z

Why did you switch to main from development that doesn't seem right

Fixed

shiv810 · 2024-09-11T05:57:28Z

@0x4007 Updated the schema, comments use voyageai for embeddings now.

0x4007

Seems generally okay. Let me see how your database looks

src/adapters/openai/helpers/embedding.ts

src/adapters/supabase/helpers/comment.ts

tests/__mocks__/adapter.ts

github-actions · 2024-09-11T11:07:54Z

Unused files (4)

src/handlers/add-issue.ts, src/handlers/delete-issues.ts, src/handlers/issue-deduplication.ts, src/handlers/update-issue.ts

shiv810 · 2024-09-11T15:57:22Z

@0x4007 Could you please check the updated changes ? Have removed OpenAI adapter, corrected the syntax and updated the dimension for the embedding column ?

shiv810 · 2024-09-13T05:04:28Z

@0x4007 Have Added Two Separate Tables as per the schema mentioned in the previous comment.

Screen.Recording.2024-09-13.at.1.01.47.AM.mov

0x4007 · 2024-09-13T05:55:35Z

supabase/migrations/20240912225853_issue_comments.sql

+    embedding Vector(1024) not null,
+    payload jsonb,
+    author_id VARCHAR not null,
+    type text not null default 'issue',


Why is the default issue on the comments table? Why even add the type column if it's in the comments table?

Its there to ignore, bot comments, and chore issues created. So, while doing vector search we can ignore such issues and comments. I can remove that, but its a feature that could be useful while performing vector search.

0x4007 · 2024-09-13T05:57:06Z

@0x4007 Have Added Two Separate Tables as per the schema mentioned in the previous comment.

Screen.Recording.2024-09-13.at.1.01.47.AM.mov

Seems mostly good but I didn't see all the headers on the first table. Does it have the payload? Just try and normalize the columns as much as you can.

Otherwise not sure why you added the type column if they are separated by type per table.

shiv810 · 2024-09-13T06:07:44Z

@0x4007 Have Added Two Separate Tables as per the schema mentioned in the previous comment.
Screen.Recording.2024-09-13.at.1.01.47.AM.mov
Seems mostly good but I didn't see all the headers on the first table. Does it have the payload? Just try and normalize the columns as much as you can.

Otherwise not sure why you added the type column if they are separated by type per table.

Yes, it includes a payload. I've retained the type column to distinguish between bot comments and bot issues. However, I can remove it if needed, as implementing that feature might be beyond the scope of this issue.

0x4007 · 2024-09-13T06:24:34Z

Yes I think its unnecessary if they are separated by type on different tables.

shiv810 · 2024-09-13T06:34:37Z

Removed the type from schema. Payload is stored for both comments and issues.

Screen.Recording.2024-09-13.at.2.31.36.AM.mov

0x4007 · 2024-09-13T06:47:15Z

Thanks for the thorough QA. You don't need to make a new video on every change! But generally when opening a pull or making major changes a video is useful.

The last idea I have (sorry for the last second changes) is to have two columns for the text plaintext and markdown

This is so we can easily do testing in the near future to compare the performance of the plaintext and markdown versions of each comment when reasoning with the LLMs. However the new ChatGPT model o1 just came out today and is supposed to be very good at reasoning, with built-in chain-of-thought reasoning capabilities. This makes me more optimistic about working with the raw markdown source code, as it provides more context (i.e. blockquotes)

Once that is implemented, you don't need to make a QA video. Just let me know and we can merge. Do that for both tables please.

…m markdown to plaintext

shiv810 · 2024-09-13T07:25:41Z

@0x4007 I have added markdown and plaintext column. o1 has a great reasoning capabilities, I think ChatGPT-Plus members have access to it already, probably will take a lot of time to be GA and be available on API.

src/adapters/utils/markdown-to-plaintext.ts

src/adapters/voyage/helpers/embedding.ts

src/handlers/add-issue.ts

0x4007 · 2024-09-13T07:35:18Z

I think ChatGPT-Plus members have access to it already, probably will take a lot of time to be GA and be available on API.

I think that only tier5 subscribers can use right now via API. I believe that we are tier4.

0x4007 · 2024-09-13T08:20:42Z

src/adapters/supabase/helpers/issues.ts

+  async findSimilarIssues(markdown: string, threshold: number): Promise<IssueType[] | null> {
+    const embedding = await this.context.adapters.voyage.embedding.createEmbedding(markdown);
+    const { data, error } = await this.supabase
+      .from("issues")
+      .select("*")
+      .eq("type", "issue")
+      .textSearch("embedding", embedding.join(","))
+      .order("embedding", { foreignTable: "issues", ascending: false })
+      .lte("embedding", threshold);
+    if (error) {
+      this.context.logger.error("Error finding similar issues", error);
+      return [];
+    }
+    return data;
+  }
+}


This seems out of scope?

This can be used for issue deduplication and stuff. I think this should be here m.

Disagree but no need to slow down this pull further.

src/adapters/voyage/helpers/embedding.ts

sshivaditya and others added 8 commits September 8, 2024 19:22

fix: config added

120a688

fix: config added

381fda1

fix: config added

95703c7

fix: config added and updated readme

32bdae9

fix: incorrect url

ca3cf2d

fix: tests

71a65a3

Merge branch 'ubiquibot:development' into development

45b3d62

fix: remove config.yml

86c56d5

shiv810 changed the base branch from development to main September 9, 2024 05:34

0x4007 requested changes Sep 9, 2024

View reviewed changes

shiv810 changed the base branch from main to development September 9, 2024 07:21

sshivaditya added 8 commits September 9, 2024 03:36

fix: nullable plaintext

8847778

fix: jest tests

5dd651e

feat: added voyage ai support

f847053

fix: cspell issue

1549feb

fix: adds serialized comment object and payload to the schema

71854b4

fix: updated handling for private repo

e6a47b4

fix: test

f8a1421

fix: cspell

bccfe23

shiv810 requested a review from 0x4007 September 11, 2024 05:57

0x4007 reviewed Sep 11, 2024

View reviewed changes

src/adapters/openai/helpers/embedding.ts Outdated Show resolved Hide resolved

src/adapters/supabase/helpers/comment.ts Outdated Show resolved Hide resolved

tests/__mocks__/adapter.ts Outdated Show resolved Hide resolved

fix: cspell, removed openai, max length of vectors is 1024'

b6a8c7c

sshivaditya added 2 commits September 11, 2024 07:10

fix: module import error

73d0a5a

fix: test

4679fe3

feat: issue dedup

6d1c521

sshivaditya added 5 commits September 12, 2024 23:25

fix: updated manifest.json

8be2379

fix: updated manifest.json

9d9197a

fix: updated manifest.json and readme.md

22679ec

fix: issue.created to issues.opened

2ea6146

fix: issue config removed updated schema

a0da267

fix: remove console.log

8d25ce1

0x4007 reviewed Sep 13, 2024

View reviewed changes

feat: added cols markdown and plaintext, adds code for conversion fro…

1a15997

…m markdown to plaintext

0x4007 reviewed Sep 13, 2024

View reviewed changes

src/adapters/utils/markdown-to-plaintext.ts Outdated Show resolved Hide resolved

0x4007 reviewed Sep 13, 2024

View reviewed changes

src/adapters/voyage/helpers/embedding.ts Outdated Show resolved Hide resolved

0x4007 reviewed Sep 13, 2024

View reviewed changes

src/handlers/add-issue.ts Outdated Show resolved Hide resolved

0x4007 reviewed Sep 13, 2024

View reviewed changes

src/handlers/add-issue.ts Outdated Show resolved Hide resolved

sshivaditya added 4 commits September 13, 2024 03:42

fix: fixed verbose to debug, changed model to voyage-large-2-instruct

a4bcc71

fix: throw error for empty body

6beefba

fix: removed custom code for conversion of markdown to plaintext

0d5d975

fix: spell check

5d6ac5a

shiv810 requested a review from 0x4007 September 13, 2024 08:01

0x4007 reviewed Sep 13, 2024

View reviewed changes

src/adapters/voyage/helpers/embedding.ts Show resolved Hide resolved

0x4007 approved these changes Sep 13, 2024

View reviewed changes

0x4007 merged commit 60c1c3e into ubiquity-os-marketplace:development Sep 13, 2024
2 checks passed

ubiquity-os bot mentioned this pull request Sep 13, 2024

Fix Database Schema #8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Database Schema #9

Fix: Database Schema #9

shiv810 commented Sep 9, 2024 •

edited

Loading

0x4007 commented Sep 9, 2024 •

edited

Loading

shiv810 commented Sep 9, 2024

shiv810 commented Sep 11, 2024

0x4007 left a comment

github-actions bot commented Sep 11, 2024 •

edited

Loading

shiv810 commented Sep 11, 2024

shiv810 commented Sep 13, 2024 •

edited

Loading

0x4007 Sep 13, 2024

shiv810 Sep 13, 2024

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024 •

edited

Loading

0x4007 Sep 13, 2024

shiv810 Sep 13, 2024

0x4007 Sep 13, 2024

Fix: Database Schema #9

Fix: Database Schema #9

Conversation

shiv810 commented Sep 9, 2024 • edited Loading

0x4007 commented Sep 9, 2024 • edited Loading

shiv810 commented Sep 9, 2024

shiv810 commented Sep 11, 2024

0x4007 left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 11, 2024 • edited Loading

Unused files (4)

shiv810 commented Sep 11, 2024

shiv810 commented Sep 13, 2024 • edited Loading

0x4007 Sep 13, 2024

Choose a reason for hiding this comment

shiv810 Sep 13, 2024

Choose a reason for hiding this comment

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024

shiv810 commented Sep 13, 2024

0x4007 commented Sep 13, 2024 • edited Loading

0x4007 Sep 13, 2024

Choose a reason for hiding this comment

shiv810 Sep 13, 2024

Choose a reason for hiding this comment

0x4007 Sep 13, 2024

Choose a reason for hiding this comment

shiv810 commented Sep 9, 2024 •

edited

Loading

0x4007 commented Sep 9, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

shiv810 commented Sep 13, 2024 •

edited

Loading

0x4007 commented Sep 13, 2024 •

edited

Loading