You need to escape the $ character, as it represents the end of the line.
str = string.gsub("String 1 $br$ String 2", "%$br%$", "\n")
If you want to capture spaces around $br$ :
str = string.gsub("String 1 $br$ String 2", "%s*%$br%$%s*", "\n")
source share