I created a small project to test Docker clustering. Basically, the cluster.sh script runs three identical containers and uses pipework to configure the bridge ( bridge1 ) on the host and add NIC ( eth1 ) to each container.
If I write to one of the containers, I can arping other containers:
# 172.17.99.1 root@d01eb56fce52 :/# arping 172.17.99.2 ARPING 172.17.99.2 42 bytes from aa:b3:98:92:0b:08 (172.17.99.2): index=0 time=1.001 sec 42 bytes from aa:b3:98:92:0b:08 (172.17.99.2): index=1 time=1.001 sec 42 bytes from aa:b3:98:92:0b:08 (172.17.99.2): index=2 time=1.001 sec 42 bytes from aa:b3:98:92:0b:08 (172.17.99.2): index=3 time=1.001 sec ^C --- 172.17.99.2 statistics --- 5 packets transmitted, 4 packets received, 20% unanswered (0 extra)
So it seems that packets can go through bridge1 .
But the problem is that I cannot ping other containers, I also cannot send any IP packets through any tools like telnet or netcat .
In contrast, the docker0 bridge and NIC eth0 work correctly in all containers.
Here is my route table
# 172.17.99.1 root@d01eb56fce52 :/# ip route default via 172.17.42.1 dev eth0 172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.17 172.17.99.0/24 dev eth1 proto kernel scope link src 172.17.99.1
and bridge configuration
# host $ brctl show bridge name bridge id STP enabled interfaces bridge1 8000.8a6b21e27ae6 no veth1pl25432 veth1pl25587 veth1pl25753 docker0 8000.56847afe9799 no veth7c87801 veth953a086 vethe575fe2
and IP FORWARD chain
# host $ sudo iptables -x -v --line-numbers -L FORWARD Chain FORWARD (policy ACCEPT 10675 packets, 640500 bytes) num pkts bytes target prot opt in out source destination 1 15018 22400195 DOCKER all -- any docker0 anywhere anywhere 2 15007 22399271 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED 3 8160 445331 ACCEPT all -- docker0 !docker0 anywhere anywhere 4 11 924 ACCEPT all -- docker0 docker0 anywhere anywhere 5 56 4704 ACCEPT all -- bridge1 bridge1 anywhere anywhere
Note that the pkts cound for rule 5 is not 0, which means that ping was correctly routed (does the FORWARD chain run after the correct routing?), But somehow did not reach the destination.
I have no idea why docker0 and bridge1 behave differently. Any suggestion?
Update 1
Here is the output of tcpdump on the target container when ping from another.
$ tcpdump -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 22:11:17.754261 IP 192.168.1.65 > 172.17.99.1: ICMP echo request, id 26443, seq 1, length 6
Note that the source IP is 192.168.1.65 , which is the host eth0 , so SNAT seems to be happening on the bridge.
Finally, a listing of the nat IP table revealed the cause of the problem:
$ sudo iptables -L -t nat ... Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 172.17.0.0/16 anywhere ...
Since my eth0 IP container is at 172.17.0.0/16 , the sent packets have the source IP address. This is why ping responses cannot return to the source.
Conclusion
The solution is to change the eth0 IP container to a different network than the standard docker0 .