, text.txt. :
sed -i 's/\(.*[^ABC]\|^\)cat\([^XYZ].*\|$\)/\1pet\2/g' text.txt
( tutorialspoint.com, man sed sed regex):
-i Edit files in place (makes backup if extension supplied)
s/???/???/ Or s/regexp/replacement/, Attempt to match regexp against the pattern space.
/ Field separator to 's'.
^ Match first character on line.
\( Start back reference.
. Match any character.
[^ABC] Do not match any charcter (^ = don't) in this list.
\| Matches regex1 or regexp2 (do not match ABC or match start of line).
\) End back reference.
cat Match cat
\1 The first back reference.
\2 The second back reference.
g Replace all matches, not just the first match.