How to filter messages before transmitting them to consumers?

Question

I am building a lead and event management system with Kafka. The problem is that we get a lot of fake potential customers (advertising). We also have many consumers in our system. Do I need to filter my ads before going to consumers? My solution is to write everything in the first topic, then read it by the filter consumer, then write it in the second topic or filter it out. But I'm not sure if it is effective or not. Any idea?

Answer 1

You can use Kafka threads ( http://kafka.apache.org/documentation.html#streamsapi ) from 0.10. + version of Kafka. This is for your use case, I think.

Answer 2

Yes - in fact, I am basically convinced that this is how you should deal with a problem in your context. Since Kafka is only useful for efficient data transfer, it can do nothing in terms of cleaning your data. Consume all the information you get from an intermediary who can run his own tests to determine what his filter is transmitting, and click on another section / section (based on your needs) to get the best data.

Answer 3

You can use Spark Streaming: https://spark.apache.org/docs/latest/streaming-kafka-integration.html .