What is the best docker + ufw practice under Ubuntu

I just tried Docker. This is awesome, but doesn't seem to work with UFW. By default, dockers manipulate iptables a bit. The result is not a mistake, not what I expected. For more details you can read Dangers of UFW + Docker

My goal is to create a system like

Host (running ufw) -> docker container 1 - nginx (as a reverse proxy) -> docker container 2 - node web 1 -> docker container 3 - node web 2 -> ....... 

I want to control incoming traffic (for example, restrict access) through ufw, so I do not want the docker to touch my iptables. Here is my test

Environment:

  • recently installed Ubuntu 14.04 (kernel: 3.13.0-53)
  • Docker 1.6.2
  • ufw redirect enabled. ( Enable UFW Forwarding )
--iptables=false been added to the Docker daemon.

First try

 docker run --name ghost -v /home/xxxx/ghost_content:/var/lib/ghost -d ghost docker run --name nginx -p 80:80 -v /home/xxxx/nginx_site_enable:/etc/nginx/conf.d:ro --link ghost:ghost -d nginx 

Bad luck. The first command is great, but the second command gives an error

 Error response from daemon: Cannot start container 

Second attempt

Then I found this: failed to associate containers with --iptables = false # 12701

After executing the following command, everything looks fine.

 sudo iptables -N DOCKER 

However, I noticed that I cannot establish outgoing connections inside containers. For example:

 xxxxg@ubuntu :~$ sudo docker exec -t -i nginx /bin/bash root@b0d33f22d3f4 :/# ping 74.125.21.147 PING 74.125.21.147 (74.125.21.147): 56 data bytes ^C--- 74.125.21.147 ping statistics --- 35 packets transmitted, 0 packets received, 100% packet loss root@b0d33f22d3f4 :/# 

If I remove --iptables=false from the Docker daemon, the Internet connection will return to normal, but ufw will not work “correctly” (well ... by my definition).

So what is docker + ufw best practice? Can anyone help?

Thanks.

Bart.

+33
docker ubuntu containers firewall iptables
source share
5 answers

I had the same problem as a few months ago, and recently I decided to describe this problem along with the solution in my blog. Here is a shortcut.

Using --iptables=false will help you little in the case you described. It is simply not enough here. By default, none of your containers can establish an outbound connection.

There is a small step that you skip on your way to have containers behind UFW here. You can use --iptables=false or create a /etc/docker/daemon.json file with the contents as follows

 { "iptables": false } 

the result will be the same, but the last option requires that you restart the entire service docker restart or even reboot if Docker had the ability to add iptables rules before you disabled this feature.

When this is done, just do two more things:

 $ sed -i -e 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/g' /etc/default/ufw $ ufw reload 

therefore, you set the default forwarding policy in UFW for acceptance and use:

 $ iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE 

Thus, you try to disable the dirty behavior of the docker in your iptables rules, and at the same time, the docker is provided with the necessary routing, so the containers do an excellent job with outgoing connections. UFW rules will still be limited from now on.

Hope this solves the problem for you and anyone who gets here looking for an answer.

I described the problem and solution in more detail at https://www.mkubaczyk.com/2017/09/05/force-docker-not-bypass-ufw-rules-ubuntu-16-04/

+29
source share

problem

This problem has existed for a long time.

Disabling iptables in Docker will lead to other problems.

Rollback changes first

If you changed your server in accordance with the current solution that we find on the Internet, first roll back these changes, including:

  • Enable Docker iptables function. Remove any changes, for example --iptables=false , including the configuration file /etc/docker/daemon.json .
  • The UFW FORWARD rule defaults back to DROP by default instead of ACCEPT .
  • Remove the rules for the Docker network in the UFW configuration file /etc/ufw/after.rules .
  • If you changed the Docker configuration files, restart Docker first. We will change the UFW configuration later, and then we can restart it.

Troubleshooting UFW and Docker

This solution should change only one UFW configuration file, all configurations and Docker options remain by default. No need to disable the Docker iptables function.

Modify the UFW configuration file /etc/ufw/after.rules and add the following rules to the end of the file:

 # BEGIN UFW AND DOCKER *filter :ufw-user-forward - [0:0] :DOCKER-USER - [0:0] -A DOCKER-USER -j RETURN -s 10.0.0.0/8 -A DOCKER-USER -j RETURN -s 172.16.0.0/12 -A DOCKER-USER -j RETURN -s 192.168.0.0/16 -A DOCKER-USER -j ufw-user-forward -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16 -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 10.0.0.0/8 -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 192.168.0.0/16 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 10.0.0.0/8 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 172.16.0.0/12 -A DOCKER-USER -j RETURN COMMIT # END UFW AND DOCKER 

Use the sudo systemctl restart ufw to restart UFW after changing the file. Now the public network cannot access any published ports of the dock, the container and the private network can visit each other regularly, and the containers can also access the external network from the inside.

If you want to allow public networks access to services provided, for example, by the Docker container, the container service port is 80 . Run the following command to allow public networks access to this service:

 ufw route allow proto tcp from any to any port 80 

This command allows the public network to access all published ports whose container port is 80.

Note. If we publish a port using the -p 8080:80 option, we should use container port 80 , not host port 8080 .

If there are several containers with service port 80, but we want the external network to have access to a specific container. For example, if the container's private address is 172.17.0.2, use the following command:

 ufw route allow proto tcp from any to 172.17.0.2 port 80 

If the network protocol of the service is UDP, for example, the DNS service, you can use the following command to allow the external network access to all published DNS services:

 ufw route allow proto udp from any to any port 53 

Similarly, if only for a specific container, such as IP address 172.17.0.2:

 ufw route allow proto udp from any to 172.17.0.2 port 53 

How does it work?

The following rules allow private networks to visit each other. Private networks are generally more reliable than public networks.

 -A DOCKER-USER -j RETURN -s 10.0.0.0/8 -A DOCKER-USER -j RETURN -s 172.16.0.0/12 -A DOCKER-USER -j RETURN -s 192.168.0.0/16 

The following rules allow UFW to determine whether public networks are allowed to visit the services provided by the Docker container. So that we can manage all the firewall rules in one place.

 -A DOCKER-USER -j ufw-user-forward 

The following rules block connection requests initiated by all public networks, but allow internal networks access to external networks. For TCP, this prevents the active establishment of a TCP connection from public networks. For UDP, all access to ports less than 32767 is blocked. Why is this port? Since UDP is stateless, it is not possible to block an acknowledgment signal that initiates a connection request, as TCP does. For GNU / Linux, we can find the range of local ports in the file /proc/sys/net/ipv4/ip_local_port_range . The default range is 32768 60999 . When accessing the UDP protocol service from a running container, the local port will be randomly selected from the range of ports, and the server will return data to this random port. Therefore, we can assume that the UDP listening port in all containers is less than 32768. This is the reason that we do not want public networks to have access to UDP ports that are less than 32768.

 -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16 -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 10.0.0.0/8 -A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 192.168.0.0/16 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 10.0.0.0/8 -A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 172.16.0.0/12 -A DOCKER-USER -j RETURN 

More

https://github.com/chaifeng/ufw-docker

 sudo wget -O /usr/local/bin/ufw-docker https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker chmod +x /usr/local/bin/ufw-docker 

using

 ufw-docker help ufw-docker install ufw-docker status ufw-docker allow webapp ufw-docker allow webapp 80 ufw-docker allow webapp 53/udp ufw-docker list webapp ufw-docker delete allow webapp 80/tcp ufw-docker delete allow webapp 

Update: 2018-09-10

Reason for choosing ufw-user-forward rather than ufw-user-input

using ufw-user-input

Pro:

Easy to use and understand, supports older versions of Ubuntu.

For example, to allow the public to visit a published port whose container port is 8080 , use the command:

 ufw allow 8080 

Against:

It not only provides container ports, but also provides host ports.

For example, if the host is running a service and the port is 8080 . The ufw allow 8080 command allows a public network to visit the service and all published ports whose container ports are 8080 . But we just want to show a service running on a host, or just a service running inside containers, not both.

To avoid this problem, we may need to use a command similar to the following for all containers:

 ufw allow proto tcp from any to 172.16.0.3 port 8080 

using ufw-user-forward

Pro:

It is not possible to expose services running on hosts and containers at the same time with the same command.

For example, if we want to publish port 8080 containers, use the following command:

 ufw route allow 8080 

The public network can access all published ports whose containers have port 8080 .

But host port 8080 still unavailable to the public network. If we want to do this, run the following command to allow port sharing on the host separately:

 ufw allow 8080 

Against:

Does not support older versions of Ubuntu, and the command is a bit more complicated. But you can use my script https://github.com/chaifeng/ufw-docker .

Conclusion

If we use an older version of Ubuntu, we can use ufw-user-input . But be careful not to expose services that should not be exposed.

If we are using a newer version of Ubuntu that supports ufw route support ufw route , we better use ufw-user-forward and ufw route to manage container firewall rules.


Update: October 6, 2018

The ufw-docker script now supports Docker Swarm. Please see the latest code for more information, https://github.com/chaifeng/ufw-docker

Set to Docker Swarm mode

We can use this script only on the manager nodes to control the firewall rules when used in Swarm mode.

  • Change all after.rules files on all sites, including managers and employees
  • Deploying this script on manager nodes

Working in Docker Swarm mode, this script will add the global service ufw-docker-agent . The chaifeng / ufw-docker-agent image is also automatically created from this project.

+43
source share

Not quite sure what your request is, but from what I can collect, you better control who can access your applications running inside Docker? I answered a similar question here to control traffic through an external proxy server, not with IP tables. Block external access to docker containers

Hope this helps

Dylan

Edit

Using the above approach, you can use UFW to allow incoming connections to port 80 (i.e., proxies). This ensures a minimum number of ports when adding an additional bonus that you can control traffic through the proxy and DNS configuration

0
source share

Why is it worth adding to @mkubaczyk answer for the case when more bridge networks are involved in the entire installation. They can be provided by Docker-Compose projects and here, how to create the right rules, given that these projects are controlled by systemd .

/etc/systemd/system/ compose-project@.service

 [Unit] Description=Docker-Compose project: %I After=docker.service BindsTo=docker.service AssertPathIsDirectory=/<projects_path>/%I AssertFileNotEmpty=/<projects_path>/%I/docker-compose.yml [Service] Type=simple Restart=always WorkingDirectory=/<projects_path>/%I ExecStartPre=/usr/bin/docker-compose up --no-start --remove-orphans ExecStartPre=+/usr/local/bin/update-iptables-for-docker-bridges ExecStart=/usr/bin/docker-compose up ExecStop=/usr/bin/docker-compose stop --timeout 30 TimeoutStopSec=30 User=<> StandardOutput=null [Install] WantedBy=multi-user.target 

/usr/local/bin/update-iptables-for-docker-bridges

 #!/bin/sh for network in $(docker network ls --filter 'driver=bridge' --quiet); do iface=$(docker network inspect --format '{{index .Options "com.docker.network.bridge.name"}}' ${network}) [ -z $iface ] && iface="br-${network}" subnet=$(docker network inspect --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' ${network}) rule="! --out-interface ${iface} --source ${subnet} --jump MASQUERADE" iptables --table nat --check POSTROUTING ${rule} || iptables --table nat --append POSTROUTING ${rule} done 

Obviously, this will not scale well.

It should also be noted that the entire basic concept will mask the source of any connection for applications running in the container.

0
source share

I spent two hours trying the suggestions above and from other posts. The only solution that worked was from Tsun's post on this Github thread:

 Append the following at the end of /etc/ufw/after.rules (replace eth0 with your external facing interface): # Put Docker behind UFW *filter :DOCKER-USER - [0:0] :ufw-user-input - [0:0] -A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A DOCKER-USER -m conntrack --ctstate INVALID -j DROP -A DOCKER-USER -i eth0 -j ufw-user-input -A DOCKER-USER -i eth0 -j DROP COMMIT And undo any and all of: Remove "iptables": "false" from /etc/docker/daemon.json Revert to DEFAULT_FORWARD_POLICY="DROP" in /etc/default/ufw Remove any docker related changes to /etc/ufw/before.rules Be sure to test that everything comes up fine after a reboot. I still believe Docker out of the box behavior is dangerous and many more people will continue to unintentionally expose internal services to the outside world due to Docker punching holes in otherwise safe iptables configs. 
0
source share

All Articles