how does cockroach solve tail latency now?

[LeGamerDc]

hi @nvanbenschoten. hope you’re doing well! I’m reaching out as I admire your expertise and wanted to ask a quick question. I noticed you once used GODEBUG=nogcassist=1 to reduce tail latency, but it seems the latest CockroachDB code has removed this. Could you share how CockroachDB currently addresses tail latency reduction? I’d really appreciate your insights. Thanks so much!

[nvanbenschoten]

I noticed you once used GODEBUG=nogcassist=1 to reduce tail latency, but it seems the latest CockroachDB code has removed this.

This is still here, though our Go runtime fork has been moved to https://github.com/cockroachdb/go/tree/ae89cf02df7a088c1f36752c0a31bfbc6a5eb5bd. If you search for gcnoassist, you'll see it, still in the same shape as when it was originally added.

Could you share how CockroachDB currently addresses tail latency reduction?

That said, we don't use this knob in production, just in tests to understand the cost of GC assist. It would be unsafe and risk OOMs since it undermines soft memory limits. Our current approach for keeping GC assist disruption down is to run with a higher default GOGC (#119605) and to minimize heap allocations throughout the system. Combined with a lot of the great work @mknyszek did in golang/go#56966 for more recent versions of Go, and this isn't nearly the problem it used to be.

I still suspect that there's more that could be done to bound the impact of GC assist to just cases where the risk of an OOM is real and looming, but that would be a large body of work in the Go runtime.

[LeGamerDc]

Thank you so much for your help and insights. They are greatly appreciated!

In fact , our workload is heavily concentrated on a single goroutine, which handles the majority of the work and memory allocation (yes, it’s for a game). So it suffers much more from the GC assist, and we are struggling ways to mitigate latency since it. Any further advice you might have would be incredibly valuable to us.

Thanks again for your time and expertise!

[mknyszek]

What GOMAXPROCS are you running with? If it's greater than say, 4, with the vast majority of allocations coming from a single goroutine, I would expect that the background GC workers should easily out-pace a single goroutine's worth of allocations, especially if they're only being produced during part of a frame.

(In a game setting, you might also be able to get away with writing your own "GC pacer" by turning the GC off and then manually triggering it with runtime.GC. This is an art to get just right, but by utilizing any idle time left in between rendering frames, you should be able to keep the GC out of latency-sensitive sections.)

I'd encourage you to follow-up on https://groups.google.com/g/golang-nuts with the output of running your program with GODEBUG=gctrace=1 and possibly a heap profile. The community there could offer suggestions.

See also my reply on the Go issue tracker. (golang/go#56966 (comment))

[LeGamerDc]

Thanks a lot !
I will have a try.