OpenGL performance: use instancing?

[lupine]

Hi,

https://code.ur.gs/lupine/ordoor is now running on my pinebook pro, and I'm investigating ways to improve performance. I think I've identified a possibility, but let me walk you through it.

The game is based on an isometric map. Each frame I draw approx. 200 "cells" in that map, each of which has 2-7 Z levels and 1-4 images, typically less than 64x64 pixels each - so perhaps 1,000 calls to screen.DrawImage, each with its own matrix to specify location on screen.

The output (prior to implementing bounds clipping, reducing resolution, and removing the ability to zoom out) looks like: https://code.ur.gs/lupine/ordoor/src/branch/master/doc/formats/img/chapter01_rendered_2018-09-08.png

All this results in just 4 draw-triangles calls per frame:

draw-triangles: dst: 5 <- src: 1, colorm: <nil>, mode clear, filter: nearest, address: clamp_to_zero
draw-triangles: dst: 5 <- src: 4, colorm: <nil>, mode source-over, filter: nearest, address: clamp_to_zero
draw-triangles: dst: 6 (screen) <- src: 1, colorm: <nil>, mode clear, filter: nearest, address: clamp_to_zero
draw-triangles: dst: 6 (screen) <- src: 5, colorm: <nil>, mode copy, filter: screen, address: clamp_to_zero

This seems pretty good to me, and this efficient automatic batching is a big part of why I switched to ebiten ^^. However, I peg a CPU core in doing this, and tend to get 10-30fps and 30-60 tps, which isn't fast enough :/. So where is the time going?

I attach a profile showing that it's spent in: a) runtime.futex (#1073 ?) - but even more of it is spent inside cgo, presumably in these glDrawElements calls: https://github.com/hajimehoshi/ebiten/blob/master/internal/graphicsdriver/opengl/context_desktop.go#L496

So what can we do about it?

My engine is drawing a limited set of sprites multiple times: one view gave me 148 unique sprites, drawn a total of 840 times. The most popular sprite was drawn 118 times per frame. The mean is 5.6, the median is 1.

Talking about this a bit on #gamedev on freenode, a suggestion made was to use glDrawElementsInstanced instead of glDrawElements: https://learnopengl.com/Advanced-OpenGL/Instancing & https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glDrawElementsInstanced.xhtml . This is available in OpenGL 3.1 and, as far as I understand it, means that the sprite drawn 118 times will become much cheaper, while the sprite drawn 1 time will be about as expensive.

Is this something you're familiar with? I don't know that it'll work, but it seems promising to me. Happy to dive into it more if you think it's worth exploring, or to talk about other ways to make this thing fast ^^.

[hajimehoshi]

Sorry but I'm not familiar with glDrawElementsInstanced, and moreover, I don't plan to update GL 2 to 3 due to backward compatibility. For example, Raspberry Pi still uses OpenGL 2. I might switch from GL to Vulkan, but there is no concrete plan yet.

From your graph, it looks like replacePixelsWithPBO seems the majority. Perhaps, are you calling ReplacePixels or NewImage every frame?

[hajimehoshi]

I think this is it: https://code.ur.gs/lupine/ordoor/src/branch/master/cmd/view-minimap/main.go#L172-L176

[lupine]

No problem, I can appreciate the desire for backwards compatibility!

I call ebiten.NewImageFromImage once the first time I use each sprite in rendering. So the first frame sees 148 calls; the second frame, zero. Sorry, I should have linked to the code: https://code.ur.gs/lupine/ordoor/src/branch/master/cmd/view-map/main.go#L192 & https://code.ur.gs/lupine/ordoor/src/branch/master/internal/assetstore/object.go#L72 ). Under the hood, they're paletted images: https://code.ur.gs/lupine/ordoor/src/branch/master/internal/data/object.go#L46

(minimap is emulating immediate-mode drawing, I haven't touched it for a while ^^).

I can eager-load the sprites to take those specific calls out of the Draw loop , which gives me this profile:

So cgo is still significant, but coming from... glBufferSubData ? Perhaps? apitrace replay --ppd tells me the following happens per frame:

frame_end
call 71613 0 0 0 0 0 0 0 0 -1 7 glBufferSubData
call 71614 0 0 0 0 0 0 0 0 -1 7 glBufferSubData
call 71615 0 0 0 0 0 0 0 0 -1 7 glBindFramebufferEXT
call 71616 0 0 0 0 0 0 0 0 -1 7 glViewport
call 71617 0 0 0 0 0 0 0 0 -1 7 glBlendFunc
call 71618 0 0 0 0 0 0 0 0 -1 3 glUseProgram
call 71619 0 0 0 0 0 0 0 0 -1 3 glUniform2fv
call 71620 0 0 0 0 0 0 0 0 -1 3 glBindTexture
call 71621 0 0 0 0 0 0 0 0 0 3 glDrawElements
call 71622 0 0 0 0 0 0 0 0 -1 3 glBlendFunc
call 71623 0 0 0 0 0 0 0 0 -1 3 glBindTexture
call 71624 0 0 0 0 0 0 0 0 0 3 glDrawElements
call 71625 0 0 0 0 0 0 0 0 -1 3 glBindFramebufferEXT
call 71626 0 0 0 0 0 0 0 0 -1 3 glViewport
call 71627 0 0 0 0 0 0 0 0 -1 3 glBlendFunc
call 71628 0 0 0 0 0 0 0 0 -1 3 glUniform2fv
call 71629 0 0 0 0 0 0 0 0 -1 3 glBindTexture
call 71630 0 0 0 0 0 0 0 0 1 3 glDrawElements
call 71631 0 0 0 0 0 0 0 0 -1 3 glViewport
call 71632 0 0 0 0 0 0 0 0 -1 3 glBlendFunc
call 71633 0 0 0 0 0 0 0 0 -1 7 glUseProgram
call 71634 0 0 0 0 0 0 0 0 -1 7 glUniform2fv
call 71635 0 0 0 0 0 0 0 0 -1 7 glUniform2fv
call 71636 0 0 0 0 0 0 0 0 -1 7 glUniform1f
call 71637 0 0 0 0 0 0 0 0 -1 7 glBindTexture
call 71638 0 0 0 0 0 0 0 0 0 7 glDrawElements
call 71639 0 0 0 0 0 0 0 0 -1 7 glFlush
frame_end

Thanks for taking the time to look at this! I don't expect you to debug my performance problems for me, so feel free to close this issue if you're certain you don't want instancing :).

[hajimehoshi]

Thank you for the more information! BufferSubData is called when sending data like vertices or indices to GPU, so calling it before DrawElements sounds reasonable.

cgo itself is not fast, but my guess is that the OpenGL driver itself is slow.

Sorry I could not give a help to you so far. If I come up with a good idea, I'll notify you.