ForceStopAsyncSend option for graceful stop in async mode #77
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JamesJJ
added a commit
to JamesJJ/moby
that referenced
this pull request
Dec 6, 2019
Signed-off-by: JamesJJ <[email protected]>
JamesJJ
changed the title
AsyncStop option for graceful stop in async modeAsyncStop option for graceful stop in async mode
JamesJJ
changed the title
AsyncStop option for graceful stop in async modeAsyncStop option for graceful stop in async mode
|
ping @tagomoris @cpuguy83 ptal (looks like this relates to moby/moby#40063) |
|
In theory this is nice, but I feel like everyone will just turn it on because close just should not block in general. Perhaps something like |
tagomoris
requested changes
Feb 12, 2020
There was a problem hiding this comment.
Basically I agree to add this feature as a new option, not to change the default behavior. IMO some people want to find processes which stuck in its shutdown sequence and resume Fluentd destination processes before terminating golang processes forcibly.
But I found some points to be improved.
@JamesJJ I'm very sorry for reviewing this change so lately, but could you check my review comments?
…de on Close() Signed-off-by: JamesJJ <[email protected]>
Signed-off-by: JamesJJ <[email protected]>
Signed-off-by: JamesJJ <[email protected]>
tagomoris
reviewed
Feb 19, 2020
Signed-off-by: JamesJJ <[email protected]>
|
LGTM. Thank you! |
tagomoris
changed the title
AsyncStop option for graceful stop in async mode
|
Merged, and tagged as v1.5.0 |
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever the target Fluentd server is down. Also the logger is currently lazily initializing the connection when it receives its first log. But this is a problem when the Fluentd server has never been available as the connection initialization blocks the select signaling the log channel should be drained.
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever the target Fluentd server is down. Also the logger is currently lazily initializing the connection when it receives its first log. But this is a problem when the Fluentd server has never been available as the connection initialization blocks the select signaling the log channel should be drained.
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever the target Fluentd server is down. Also the logger is currently lazily initializing the connection when it receives its first log. But this is a problem when the Fluentd server has never been available as the connection initialization blocks the select signaling the log channel should be drained.
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs.
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063.
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 22, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 23, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 23, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 23, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Apr 24, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting for tcp timeout. To fix these edge cases, the connection isn't initialized lazily anymore. However, it's still initialized asynchronously and with the exponential back-off retry. The Fluent.run() method has been adapted to wait for either the connection to become ready or to receive a stop signal on the stopRunning channel before starting to unqueue logs. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
May 21, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After() and a new stopAsyncConnect channel has been introduced to stop it. Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Aug 4, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After() and a new stopAsyncConnect channel has been introduced to stop it. Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Nov 11, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After() and a new stopAsyncConnect channel has been introduced to stop it. Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Nov 11, 2020
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After(). Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. The associated cancel function is stored in Fluent and got called by Close() when ForceStopAsyncSend option is enabled. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Oct 19, 2021
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After(). Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. The associated cancel function is stored in Fluent and got called by Close() when ForceStopAsyncSend option is enabled. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
pushed a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Oct 21, 2021
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After(). Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. The associated cancel function is stored in Fluent and got called by Close() when ForceStopAsyncSend option is enabled. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
added a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Oct 21, 2021
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After(). Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. The associated cancel function is stored in Fluent and got called by Close() when ForceStopAsyncSend option is enabled. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
akerouanton
added a commit
to akerouanton/fluent-logger-golang
that referenced
this pull request
Oct 28, 2021
PR fluent#77 introduced a new parameter named ForceStopAsyncSend. It can be used to tell the logger to not try to send all the log messages in its buffer before closing. Without this parameter, the logger hangs out whenever it has logs to write and the target Fluentd server is down. But this new parameter is not enough: the connection is currently lazily initialized when the logger receive its first message. This blocks the select reading messages from the queue (until the connection is ready). Moreover, the connection dialing uses an exponential back-off retry. Because of that, the logger won't look for messages on `stopRunning` channel (the channel introduced by PR fluent#77), either because it's blocked by the Sleep used for the retry or because the connection dialing is waiting until dialing timeout (eg. TCP timeout). To fix these edge cases, the time.Sleep() used for back-off retry has been transformed into a time.After(). Moreover, the dialer.Dial() call used to connect to Fluentd has been replaced with dialer.DialContext() and a cancellable context is now used to stop the call to that method. The associated cancel function is stored in Fluent and got called by Close() when ForceStopAsyncSend option is enabled. Finally, the Fluent.run() method has been adapted to wait for both new messages on f.pending and stop signal sent to f.stopRunning channel. Previously, both channels were selected independently, in a procedural fashion. This fix is motivated by the issue described in: moby/moby#40063. Signed-off-by: Albin Kerouanton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As I understand, existing behaviour in Async mode is:
MaxRetrytimes, and if still not successfully sent the event is discardedClose()is called, the pending event channel is closed and attempting to add any new events will return an errorClose()is called, it will block until all remaining events in the buffer have either been successfully sent, or each one has run through it'sMaxRetrynumber of failures. This means thatClose()can block for a very long time if the number of unsent events is large.This PR introduces a new option
AsyncStop. When true in Async mode and Close() is called, the current write() / MaxRetries iteration will complete for the single event in-progress, and then any remaining events in the buffer will be discarded. This may help to significantly reduce the time thatClose()blocks when fluentd is continuously available.