Docking Consul, Zouker and Kafka at Amazon-ECS

Question

I am trying to deploy Kafka dockers (with Zookeeper) in AWS. I cleared some resources and walked over. But when I go into my Kafka boxes in AWS, I see that Kafka comes out with -1. I can only assume that Kafka cannot reach Zookeeper because I did not create any name resolution. I see the Consul as a solution to this (see here ).

It looks like a decent resource. But I can’t even successfully launch the official consul image of dockers (in server or agent mode). I can get this one to work (here used here ). But I like to use an official look, if possible.

Has anyone received the official image of the Consul working with Kafka and Zoopiter?

A) To start, I think Id would like to make it work in docker. This is my docker-compose.yml file when I try to use Kafka for the consul to open the Zookeeper nodes.

Below is the corresponding code snippet that you can run with docker-compose up consul . This does not show any errors. But I can not reach http: // localhost: 8500 . And ii) How can I then indicate Kafka to the customer Zookeeper and Kafka Kafka?

 version: '2' services: consul: image: consul:0.8.3 expose: - 8300 - 8301 - 8301/udp - 8302 - 8302/udp - 8400 - 8500 - 8600 - 8600/udp ports: - 8300:8300 - 8301:8301 - 8301/udp:8301/udp - 8302:8302 - 8302/udp:8302/udp - 8400:8400 - 8500:8500 - 8600:8600 - 8600/udp:8600/udp command: "agent -server -bind=0.0.0.0 -retry-join=0.0.0.0 -bootstrap-expect=1" 

B) Then I can try to use it in AWS (via Terraform ). Ideally, an Id should have 10 Kafka nodes and 2 Zookeeper nodes. But this is like a fast-moving goal.

EDIT

As @MattSchuchard pointed out, there is the Mesos option, with Marathon and Chronos.

A) But I wanted to try something that, in my opinion, would be simpler (i.e. Consul, Zoopiter, Kafka) before stratifying into another abstraction. Installing DCOS seems simplified by things like zutherb / terraform-dcos .

B) But moving to this route would mean changing the deployment workflow: i) just deploying ECS ​​instances, ii) deploying a DCOS cluster for AWS, and then deploying docker images. If I could know that it was easy to deploy Consul, Zookeeper, Kafka, etc., Containers in a marathon, then that would be good. As far as I can tell, Id should use docs cli, losing the declarative Terraform property for my real applications and services. Since in its current form there is no official Terraform provider for interaction with Marathon ( nicgrayson / terraform-provider-marathon , despite this).

C) So, let's look at DCOS Mesosphere. But hopefully theres just a small configuration error with what I have now.

Answer 1

Has anyone received the official image of the Consul with Kafka and Zoopiter?

Yes, I have :) See https://github.com/dmstr/docker-roj/blob/master/data/example/discovery/consul/docker-compose.yml for an example config. Despite the mapped ports, I use host network mode for the consul, which just worked better.

You can take a look at the docs on how to set up a swarm with a consul on AWS .

Although this may not answer all your questions, it may be a good resource to see how you can customize your swarm; as this is a fairly simple solution. Feel free to send questions or PR to roj .