Using Sed to Extend Environment Variables Inside Files

Question

I would like to use Sed to expand variables inside a file.

Suppose I exported the variable VARIABLE = something and have a “test” file with the following:

I'd like to expand this: "${VARIABLE}" 

I tried commands like the following, but to no avail:

 cat test | sed -e "s/\(\${[AZ]*}\)/`eval "echo '\1'"`/" > outputfile 

The result is an “output file” with an unchanged variable:

 I'd like to expand this: "${VARIABLE}" 

However, running eval "echo '${VARIABLE}' in the bash console causes the value" something "to be reflected. In addition, I tested and this template is trully matched.

The desired result will be

 I'd like to expand this: "something" 

Can anyone shed some light on this?

Answer 1

View your trial version:

 cat test | sed -e "s/\(\${[AZ]*}\)/`eval "echo '\1'"`/" > outputfile 

The reason this does not work is because it requires foresight from the shell. The sed script command is created before any pattern is matched by sed, so the shell will not be able to do the job for you.

I have done this a few ways in the past. Usually I have a list of known variables and their values, and I made a substitution from this list:

 for var in PATH VARIABLE USERNAME do echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g' done > sed.script cat test | sed -f sed.script > outputfile 

If you want to arbitrarily map variables, you must either process the entire environment (instead of using a fixed list of variable names, use the result from env , edit accordingly), or use Perl or Python.

Note that if the environment variable value contains a slash in your version, you will run into problems using a slash as a field separator in the s /// notation. I used "%" since a relatively small number of environment variables use this, but some of them are found on some machines that contain the characters "%", and therefore the complete solution is more complicated. You also need to worry about the backslash in value. You probably need to use something like " $(eval echo "\$$var" | sed 's/[\%]/\\&/g') to avoid backslashes and percent characters in the variable value surroundings. Final wrinkle: some sed versions had (or had) limited capacity for script size - older versions of HP-UX had a limit of about 100. I'm not sure if this is still a problem, but it was only 5 years ago.

The simplest adaptation of the original script reads:

 env | sed 's/=.*//' | while read var do echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g' done > sed.script cat test | sed -f sed.script > outputfile 

However, the best solution is using the fact that you already have values ​​in the output from env , so we can write:

 env | sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script cat test | sed -f sed.script > outputfile 

This is safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with the shell metacharacters in the variable values. I think that this version can only be launched if there are any problems with exiting env .

Beware - writing sed scripts with sed is an esoteric lesson, but it illustrates the power of good tools.

All of these examples do not account for the temporary file (s).

Answer 2

Perhaps you can do without using sed:

 $ echo $VARIABLE something $ cat test I'd like to expand this: ${VARIABLE} $ eval "echo \"`cat test`\"" > outputfile $ cat outputfile I'd like to expand this: something 

Let the interpolation of shell variables do the job.