Skip to content

How to force a complete restart of a projection #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
bartelink opened this issue Jan 16, 2019 · 8 comments
Open

How to force a complete restart of a projection #123

bartelink opened this issue Jan 16, 2019 · 8 comments

Comments

@bartelink
Copy link

bartelink commented Jan 16, 2019 •
edited
Loading

I'm looking to restart a set of processing from the dawn of time [after processing for some time with a given prefix], discarding all progress (as opposed to the case discussed in #84 where one is indicatign the starting position one would like for a new lease collection (StartFromBeginning).

From my research, it seems:

  • If I mint a new prefix, I seem to see all the documents (as one'd expect)
  • if I use an existing lease, I can't seem to find any way via the API surface area to tell the processor I want to really Re start From The Beginning
  • There is a deprecated DiscardExistingLeases thing which sounds like what I want to achieve

Workarounds that spring to mind for me:

  1. maintain generation externally, and bump that when I want to trigger reprocessing (and include that in the LeasePrefix)
  2. perform surgery on the lease collection manually and/or with hack code as the API does not offer a way to achieve this

Am I missing something? If the only way I'm going to be able to achieve this effect is by doing one of the two things that spring to mind for me, then the xmldocs need updating to indicate that this feature does not do what it sounds like (either that, or if you can explain what use case this does fulfil, that'd work for me too)
@bartelink bartelink mentioned this issue Jan 16, 2019
Closed
@bartelink bartelink changed the title How To Cause Reprocessing from the start How to force a complete restart of a projection Jan 16, 2019
This was referenced Jan 16, 2019
Closed
Merged
@ealsur
Copy link
Member

ealsur commented Jan 18, 2019

@kadukf @mkolt Do we know why the original DiscardExistingLeases was deprecated?

@mkolt
Copy link
Contributor

mkolt commented Jan 18, 2019 via email

@bartelink
Copy link
Author

I'm not suggesting it should be equally prominent in the builder DSL or a general options record - clearly a destructive thing like this should not share space in the docs with stuff that's actually talking about a semantic defining some consistency one is expecting.

I think my ideal surface-level API in this space would be be to have a top level Method with something like Destroy or Dangerous in the name.

It's definitely important to have to have such a function though, whether that be in the form of

  • a "Task DiscardAllProgress() on the processor as a first class thing
  • a separated one-off call which intentionally lives somewhere else given how you might be pulling the rug out from sister-projectors in a cluster.

Having that also allows xmldocs like the one that prompted #84 to point somewhere as a counterpoint to underscore the purpose of something.

This was referenced Jan 18, 2019
Open
Closed
@jsmithtx
Copy link

@bartelink Why not just delete your lease and set the StartFromBeginning = true? As long as you've named your leases using the LeasePrefix option, it is straight-forward to find the two docs that must be deleted to wipe out a lease.

@bartelink
Copy link
Author

@jiffypopjr Yes, I could - the reason I'm asking here though is:

a) there was once such a facility
b) others demonstrably need it
c) not having it makes the API confusing and incomplete

Hence I'm seeking some insight as to whether there are plans to reinstate and/or whether e.g. a PR to add such a facility in any manner would be admitted.

@bartelink
Copy link
Author

@ealsur @kadukf @mkolt right now, I'm looking at having to build something to delete the state which will need to stay in sync with the CFP impl.

If I am to do this, I'd much prefer to do that as a PR here - I'd really appreciate some information as to whether such a PR would be accepted and/or whether there is a plan to provide this at any point.

While there is room to argue that if one wanted to reproject from the start, that one should version/burn the LeaseId for this projection it would not reflect the intention - I want to
a) discard the progress accumulated by this lease-set and
b) put it beyond use
c) have a guarantee that each processor will restart

The other point I'll make again, is that, while aspect c is hard to deliver, not having this facility makes it harder to understand the API for someone thinking of this as being like any other projector/consumer progress thing out there; thus if the answer is "no, this package will not provide such a facility for the foreseeable future", I'd be at least insisting on the readme.md mentioning this as being out of scope.

I'm intending to keep this issue open until there's a clear response on this - I need to provide a clear answer as to how to accomplish the standard "blow away the projected state and replay" requirement given I'm using an abstraction that attempts to encapsulate such concerns in full.

@mkolt
Copy link
Contributor

mkolt commented Feb 7, 2019 via email

@bartelink
Copy link
Author

@mkolt Yes, that sounds like it would fit

@bartelink bartelink mentioned this issue Mar 10, 2019
Open
@bartelink bartelink mentioned this issue Jul 2, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bartelink @ealsur @jsmithtx @mkolt