Large Manifest file is repeatedly re-created

[mmontour1306]

In a large or poorly-tuned database with a very large number of files, the MANIFEST-nnnn file can grow bigger than the MaxManifestFileSize option. During compaction there are repeated calls to the commit() function in session.go which contains the code:

	if s.manifest == nil || s.manifest.Size() >= s.o.GetMaxManifestFileSize() {
		// manifest journal writer not yet created, create one
		err = s.newManifest(r, nv)
	} else {
		err = s.flushManifest(r)
	}

When the s.newManifest() replacement is unable to reclaim a meaningful amount of space, there will be a call to s.newManifest() on every call to commit(). This in turn causes a large performance degradation of the overall compaction process.

At a minimum the code could check s.manifest.Size() immediately following the s.newManifest() call and log an error message if it was already above s.o.GetMaxManifestFileSize().

Another option would be to store the size of the newly-created manifest in the session struct and then use this as a dynamic limit to avoid calling the s.newManifest() branch again until the size had grown (say) 10% beyond its previous value.

[patrick-ogrady]

I think this may be fixed in #409.

[mmontour1306]

#409 doesn't fix this issue. I applied a version of that change with some logging:

	if s.manifest == nil {
		// manifest journal writer not yet created, create one
		err = s.newManifest(r, nv)
	} else if s.manifest.Size() >= s.o.GetMaxManifestFileSize() {
		s.logf("DEBUG Before size %d limit %d", s.manifest.Size(),  s.o.GetMaxManifestFileSize())
		// pass nil sessionRecord to avoid over-reference table file
		err = s.newManifest(nil, nv)
		s.logf("DEBUG After size %d", s.manifest.Size())
	} else {
		err = s.flushManifest(r)
	}

The result still shows a newly-created Manifest larger than the maximum size, triggering that code branch over and over again:

14:22:53.095908 DEBUG Before size 173084363 limit 67108864
14:22:53.724776 DEBUG After size 173083903
14:22:53.724803 table@compaction committed F-5 S-6KiB Ke·0 D·16 T·784.956107ms
14:22:53.724830 table@compaction L4·1 -> L5·6 S·14MiB Q·28212574516
14:22:53.724921 table@remove removed @79873147
14:22:53.724948 table@remove removed @79873148
14:22:53.725445 table@remove removed @84289218
14:22:53.725468 table@remove removed @66775554
14:22:53.725483 table@remove removed @66775556
14:22:53.725497 table@remove removed @66775557
14:22:53.725512 table@remove removed @66775558
14:22:53.725526 table@remove removed @66775560
14:22:53.747345 table@build created L5@84289224 N·21313 S·8MiB 0xbd84f684:0xbd850827
14:22:53.762720 table@build created L5@84289225 N·16107 S·6MiB 0xbd850827:0xbd851650
14:22:53.872280 version@stat F·[2 1809 13921 34052 171610 1658425] S·7TiB[202MiB 10GiB 100GiB 242GiB 250GiB 7TiB] Sc·[2.00 5.13 5.13 0.25 6.41 0.39]
14:22:53.872304 DEBUG Before size 173083903 limit 67108864
14:22:54.510743 DEBUG After size 173083443
14:22:54.510789 table@compaction committed F-5 S-6KiB Ke·0 D·12 T·785.935213ms
14:22:54.510813 table@compaction L4·1 -> L5·1 S·6MiB Q·28212574516
14:22:54.513721 table@remove removed @79873149
14:22:54.513754 table@remove removed @66775561
14:22:54.513782 table@remove removed @66775573
14:22:54.513799 table@remove removed @66775574
14:22:54.513820 table@remove removed @66775575
14:22:54.513839 table@remove removed @66775577
14:22:54.513854 table@remove removed @66775578
14:22:54.526575 table@build created L5@84289227 N·17038 S·6MiB 0xbd850827:0xbd851650
14:22:54.652267 version@stat F·[2 1809 13921 34052 171609 1658425] S·7TiB[202MiB 10GiB 100GiB 242GiB 250GiB 7TiB] Sc·[2.00 5.13 5.13 0.25 6.41 0.39]
14:22:54.652286 DEBUG Before size 173083443 limit 67108864
14:22:55.196077 db@close closing
14:22:55.270263 DEBUG After size 173083352
14:22:55.270287 table@commit exiting
14:22:55.270404 db@close done T·74.326254ms

By the way, this DB is from a partially synchronized Ethereum archive node which I'm presently converting to a larger options.CompactionTableSize. It is possible to work around the issue by specifying a larger MaxManifestFileSize in the geth ethdb/leveldb/leveldb.go file.