storageContext with vectorStore not saving refDocInfo

[gorango]

Consider the following two configs.

Using pesistDir with the default SimpleVectorStore:

const storageContext = await storageContextFromDefaults({
	persistDir: './storage'
})
const document = new Document({ text: 'Test test', metadata: { foo: 'bar' } })
const index = await VectorStoreIndex.fromDocuments([document], { storageContext })

Using persistDir with PGVectorStore (or QdrantVectorStore):

const storageContext = await storageContextFromDefaults({
	persistDir: './storage',
	vectorStore: new PGVectorStore(),
})
const document = new Document({ text: 'Test test', metadata: { foo: 'bar' } })
const index = await VectorStoreIndex.fromDocuments([document], { storageContext })

---

The vectorStore variant will produce a doc_store.json without a "docstore/ref_doc_info" key:

{
	"docstore/data": {
		"b73b8d28-db69-4701-8d8f-6ea76bb39c7f": {
			"__data__": {
				"id_": "b73b8d28-db69-4701-8d8f-6ea76bb39c7f",
				"metadata": {
					"foo": "bar"
				},
				"excludedEmbedMetadataKeys": [],
				"excludedLlmMetadataKeys": [],
				"relationships": {},
				"text": "Test test",
				"textTemplate": "",
				"metadataSeparator": "\n",
				"type": "DOCUMENT",
				"hash": "UgWCYBNRISLyeW2KUXqROo2xkEI1rZaNNA0tWzmMIGU="
			},
			"__type__": "DOCUMENT"
		}
	},
	"docstore/metadata": {
		"b73b8d28-db69-4701-8d8f-6ea76bb39c7f": {
			"docHash": "UgWCYBNRISLyeW2KUXqROo2xkEI1rZaNNA0tWzmMIGU="
		}
	}
}

Whereas the first variant produces a much more elaborate doc_store.json:

{
	"docstore/data": {
		"457c098c-b2fb-4e6a-b449-b694f0a7b65b": {
			"__data__": {
				"id_": "457c098c-b2fb-4e6a-b449-b694f0a7b65b",
				"metadata": {
					"foo": "bar"
				},
				"excludedEmbedMetadataKeys": [],
				"excludedLlmMetadataKeys": [],
				"relationships": {},
				"text": "Test test",
				"textTemplate": "",
				"metadataSeparator": "\n",
				"type": "DOCUMENT",
				"hash": "UgWCYBNRISLyeW2KUXqROo2xkEI1rZaNNA0tWzmMIGU="
			},
			"__type__": "DOCUMENT"
		},
		"bd6aef20-4800-4f81-9418-887688aea46f": {
			"__data__": {
				"id_": "bd6aef20-4800-4f81-9418-887688aea46f",
				"metadata": {
					"foo": "bar"
				},
				"excludedEmbedMetadataKeys": [],
				"excludedLlmMetadataKeys": [],
				"relationships": {
					"SOURCE": {
						"nodeId": "457c098c-b2fb-4e6a-b449-b694f0a7b65b",
						"metadata": {
							"foo": "bar"
						},
						"hash": "UgWCYBNRISLyeW2KUXqROo2xkEI1rZaNNA0tWzmMIGU="
					}
				},
				"text": "Test test",
				"textTemplate": "",
				"endCharIdx": 9,
				"metadataSeparator": "\n",
				"type": "TEXT",
				"hash": "ETm4T1kLNQ0isqrvyK+OMifuvN5TDFiOh9ENpbZ+m8I="
			},
			"__type__": "TEXT"
		}
	},
	"docstore/metadata": {
		"457c098c-b2fb-4e6a-b449-b694f0a7b65b": {
			"docHash": "UgWCYBNRISLyeW2KUXqROo2xkEI1rZaNNA0tWzmMIGU="
		},
		"bd6aef20-4800-4f81-9418-887688aea46f": {
			"docHash": "ETm4T1kLNQ0isqrvyK+OMifuvN5TDFiOh9ENpbZ+m8I=",
			"refDocId": "457c098c-b2fb-4e6a-b449-b694f0a7b65b"
		}
	},
	"docstore/ref_doc_info": {
		"457c098c-b2fb-4e6a-b449-b694f0a7b65b": {
			"nodeIds": [
				"bd6aef20-4800-4f81-9418-887688aea46f"
			],
			"extraInfo": {}
		}
	}
}

This means that calling docStore.getAllRefDocInfo() only works with a default vectorStore (for querying the stored docs).

So I'm wondering if this is by design and if it hinders the performance of retrievers and/or queryEngines? Should I submit this as an issue? I spent several hours trying to trace the calls to figure out the reason for this discrepancy without any luck. Would love to be able to understand what is happening here.

[himself65]

I'm guessing this is a issue that pg vector store didn't handle stuff correctly

[gorango]

I figured it out. The answer was in the VectorStoreIndex.insertNodesToStore where the NOTE indicates that "if the vector store doesn't store text, we need to add the nodes to the index struct and document store... if the vector store keeps text, we only need to add image and index nodes"

In the meantime, I confirmed that all other functionality is identical between the vector stores, so this can be closed.