NRG: Drop append entries when upper layer is overloaded #4735
Conversation
neilalexander
force-pushed
the
neil/raftoverload
branch
from
d907c97
to
d361851
Compare
neilalexander
force-pushed
the
neil/raftoverload
branch
from
d361851
to
c538f38
Compare
server/raft.go
Outdated
|
|
||
| // Updates the overloaded state. Lock must be held. | ||
| func (n *raft) updateOverloadState() { | ||
| n.overload.Store(n.apply.len() >= overloadThreshold || n.commit-n.applied >= uint64(overloadThreshold)) |
There was a problem hiding this comment.
More about memory then number of elements I think, WDYT?
|
|
||
| // Pushes to the apply queue and updates the overloaded state. Lock must be held. | ||
| func (n *raft) pushToApply(ce *CommittedEntry) { | ||
| n.apply.push(ce) |
There was a problem hiding this comment.
I think this function returns pending, so could short circuit some work here.
| // If we are overwhelmed, i.e. the upper layer is not applying entries | ||
| // fast enough and our apply queue is building up, start to drop new | ||
| // append entries instead. | ||
| if n.Overloaded() { |
There was a problem hiding this comment.
Just check n.apply.len()? Always present, so do not need to lock the raft group, and ipq underneath has its own which we use here anyway.
There was a problem hiding this comment.
Overloaded() doesn't take a lock but now just checks the apply queue size.
There was a problem hiding this comment.
It takes the ipq lock, so we are now taking it twice in that function I think.
neilalexander
force-pushed
the
neil/raftoverload
branch
from
c538f38
to
358df16
Compare
| // If we are overwhelmed, i.e. the upper layer is not applying entries | ||
| // fast enough and our apply queue is building up, start to drop new | ||
| // append entries instead. | ||
| if n.Overloaded() { |
There was a problem hiding this comment.
It takes the ipq lock, so we are now taking it twice in that function I think.
|
Haven't benched it yet but I will do so. |
neilalexander
changed the title
neilalexander
force-pushed
the
neil/raftoverload
branch
from
1817cd4
to
19181b5
Compare
Signed-off-by: Neil Twigg <[email protected]>
neilalexander
force-pushed
the
neil/raftoverload
branch
from
19181b5
to
9953f1b
Compare
This adds a mechanism for the Raft layer to detect when the upper layer isn't keeping up with applies. When the overloaded condition is detected, we'll start dropping newly incoming append entries instead so that the apply queue doesn't build up endlessly.
Still need to figure out what threshold makes sense here, or how it makes sense to configure it.
Signed-off-by: Neil Twigg [email protected]