Stream failure doesn't appear to restart the lease

[jsmithtx]

We are having an issue where stream processing failures are throwing an exception, restarting the service, appear to reacquire the lease, but the stream basically goes into a mode where it appears alive but nothing is ever received. It may be that if the restart occurs inside a certain time window, it thinks the lease is normal, but it ends up in this weird "zombie" mode where nothing is received. Our solution for now has been to restart the service again, which seems to work, suggesting it may be a timing issue.

Any suggestions for how to deal with this appropriately?

I have listed the logs below:

21:04:42 [Information] Partition "0": renewed lease with result True
21:04:50 [Information] Host '"host-4b8033ae-ae23-409b-9e97-2d2c4963eb77"' 1 partitions, 1 hosts, 0 available leases, target = 1, min = 0, max = 0, mine = 1, will try to take 0 lease(s) for myself'.

-- exception --

21:04:54 [Information] Stopping processor...
21:04:54 [Information] Partition load balancer task stopped.

-- service restarted -- (notice there is no acquire lease info logs though!)

21:04:59 [Information] Partition "0": renewed lease with result True
21:05:16 [Information] Partition "0": renewed lease with result True
21:05:33 [Information] Partition "0": renewed lease with result True
21:05:50 [Information] Partition "0": renewed lease with result True
21:06:07 [Information] Partition "0": renewed lease with result True
21:06:24 [Information] Partition "0": renewed lease with result True

[jsmithtx]

I have created a small repro project with instructions in the readme that demonstrates the problem - this is a major blocker for us, so if someone has some insight, that is greatly appreciated!

https://github.com/AMGHDev/eventstream-reader

[jsmithtx]

I finally figured out how to ensure the processor is stopped... in the sample above, Line 59 of the file below must have a ConfigureAwait(false). It appears there is a deadlock somewhere when the processor stops and so it never stops unless the ConfigureAwait is added.

https://github.com/AMGHDev/eventstream-reader/blob/master/ChangeFeedConsole/SampleListener.cs