Kafka Storm spout changes topology and consumes from the old offset

Question

I use kafka spout to consume messages. But in case I need to change the topology and load, will it come back from the old message or start with a new message? Kafka Spout gives us a timestamp specification from which to consume, but how can I find out a timestamp?

Answer 1

If you are using KafkaSpout, make sure of the following:

• In your SpoutConfig, "id" and "zkroot" do NOT change after redistributing a new version of the topology. The storm uses "zkroot", "id" to save the theme offset in zookeeper
• KafkaConfig.forceFromStart false.

KafkaSpout zookeeper. , forceFromStart true ( , ) KafkaConfig KafkaSpout, zookeeper. , false.

, KafkaConfig.forceFromStart , topologys main(). , Kafka .

Answer 2

spoutConfig.forceStartOffsetTime(-1);

, , . , -1, , -2.

Answer 3

:

• , :

  forceFromStart = true
  
  startOffsetTime = -2
  

. , forceFromStart startOffsetTime , , , zookeeper.

, zookeeper . , zookeeper.

2. , , ​​ , :

   forceFromStart = false
   

, startOffsetTime zookeeper, .

, , , .

3. / , :

   forceFromStart = true
   
   startOffsetTime = -1
   

, zookeeper , .