I think you answered your own question by saying "you would never be too safe"
The fact that web.config cannot be viewed or downloaded does not guarantee that an attack within your organization cannot occur. Someone with the right of access may violate security and receive data if they are visible in public. Although, as you said, an attack through Http is unlikely to get this file, others who have the correct access can be accessed using other means, such as remote access to files, remote desktop, etc. Encryption + assigning the correct access / authorization makes it difficult for an attacker to try to break your system.
In safety, you just have to be paranoid and take all the measures that you think are possible to make the data as safe as possible, and your work gives the attacker a hard time to get any information. I donβt think there is such a thing as 100% security. In a sense, you will always be vulnerable to a security threat, all we can do is try to minimize it as much as possible.
source share