Use recyclable & paginated buffers for search hits

[DaveCTurner]

Today SearchHit#source is a plain BytesReference which is probably inappropriate since hits are occasionally large and often quite short-lived. For instance, if fetching hits from a remote node, the coordinator will read this field into a single new byte[] allocated within here:

elasticsearch/server/src/main/java/org/elasticsearch/search/SearchHit.java

Line 152 in 1925712

source = in.readBytesReference();

Those allocations can be humongous and will impose quite some load on the GC. We should instead consider move to using recyclable and paginated buffers for search hits. In terms of network reads that means using StreamInput#readReleasableBytesReference() and anywhere we're allocating it locally we should use the node's BigArrays. Unfortunately that has quite wide-reaching consequences, since it would mean that SearchHit must now become Releasable, as must all its owning classes, so that the buffers can be recycled once they're no longer in use.

This has substantial overlap with #89656 since a large part of the work involved in tracking this memory usage for circuit-breaking purposes would also require us to add such a lifecycle to SearchHit and its owners.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[iverase]

Those source bytes are normally compressed, so the situation becomes more difficult when we decompress it, as it generates yet another bigger plain BytesReference:

elasticsearch/server/src/main/java/org/elasticsearch/search/SearchHit.java

Line 379 in 795e07c

this.source = CompressorFactory.uncompressIfNeeded(this.source);

[DaveCTurner]

That's true, but it isn't hard to decompress a recyclable BytesReference into another recyclable BytesReference too as long as we can add the plumbing to make the recycler available where it's needed.

[iverase]

What will be harder is that those plain bytes come all the way from lucene:

https://github.com/apache/lucene/blob/008a0d420635239570e09e0862a357f512781c60/lucene/core/src/java/org/apache/lucene/codecs/lucene90/compressing/Lucene90CompressingStoredFieldsReader.java#L243

I hope we can tell lucene to give as a reference on the shape of a DataInput or similar (like in this proposal apache/lucene#12460)?

[DaveCTurner]

I hope we can tell lucene to give as a reference on the shape of a DataInput or similar

Yeah that's certainly another place worth fixing. But the problems in this area are particularly noticeable on coordinating nodes today, so fixing this in the transport layer will already help. And the work to give SearchHit a proper lifecycle will pay dividends in various other ways too.

[benwtrent]

@original-brownbear can we close this?

[javanna]

What's left to do (credits to @original-brownbear ): remove all remaining uses of SearchHit#asUnpooled if we wanted to be thorough + make Lucene read sources to pooled buffers as well

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)