Geek Answers Handbook

Best practices for draining or cleaning a pubsub Google Cloud theme

For pubsub topics with a post count of ~ 100k, what is the best practice for draining / deleting / cleaning / deleting all posts using the gcloud-java SDK?

Possible solutions:

  • Removing and re-creating subscribers and then publishers

  • High concurrency pull + ack (easy to get into this quota)

  • Something else

I hope this process can be fast (no more than 60 seconds, say), reliable and uses supported SDK methods with minimal other code.

+29
google-cloud-pubsub
source share
2 answers

Update describing the snapshot and search function: You can use seek in the Pub / Sub subscription (currently in alpha format) to receive older messages, trying to find the timestamp that matches the current one. The best way is to use the gcloud command line tool . The command to confirm messages published before a certain timestamp will be as follows:

gcloud pubsub subscriptions seek <subscription path> --time=yyyy-mm-ddThh:mm:ss

Previous answer before adding the snapshot and searching: Currently, Google Cloud Pub / Sub is not able to clear old messages, although we want to add it. Removing and re-creating a subscription will be the most effective way to clean it, both in terms of time and cost. You do not have to do anything with your publishers; Any messages posted at the point after re-creation will be sent to subscribers with the re-created subscription.

+41
source share

.seek () did not work very well in python, using the provided Timestamp () instantiated from the timestamp class. In my tests, the posting time of the post clearly showed the time before I use .seek (); But the messages until the time I'm looking for do not confirm everything (which will deplete the message in the subscription). In any case, many messages have returned from the dead!

The best way I can find now is to run a cleanup function using synchronous extraction (subscriber.pull), with some sane max_messages and timeout; or with asynchronous voltage, but give it a longer wait time (for example, 6-8 + seconds); unfortunately, if you do not disable synchronous extraction, it will simply hang and wait if it fails at least one message; and for asynchronous it will just wait forever if you don't time out.

I would like to try a search on a snapshot, but did not have the necessary permissions (also to delete / recreate a subscription)

0
source share

More articles:


All Articles