How to replace $ {} placeholders in a text file?

Question

I want to transfer the output of the "template" file to MySQL, a file with variables like ${dbName} interspersed. What is a command line utility to replace these instances and dump the output to standard output?

Answer 1

Sed !

This file template.txt:

 The number is $ {i}
 The word is $ {word}

we just have to say:

 sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt 

Thanks to Jonathan Leffler for the prompt to pass a few arguments -e to the same sed call.

Answer 2

Update

Here is a solution from yottatsa on a similar question, which replaces only variables like $ VAR or $ {VAR}, and is a short one-line

 i=32 word=foo envsubst < template.txt 

Of course, if I and the word are in your environment, then it's just

 envsubst < template.txt 

On my Mac, it looks like it was installed as part of gettext and from MacGPG2

Old answer

Below is an improvement on mogsie 's solution on a similar issue. My solution does not require you to escalate double quotes, mogsie, but he is one liner!

 eval "cat <<EOF $(<template.txt) EOF " 2> /dev/null 

The strength of these two solutions is that you only get a few types of shell extensions that are not usually found in $ ((...)), `...` and $ (...), although the backslash here is escape -symbol, but you don’t have to worry about the parsing error, and it makes a few lines just fine.

Answer 3

Use /bin/sh . Create a small shell script that sets the variables and then parses the template using the shell. Also (edit the correct processing of translation strings):

Template.txt file:

 the number is ${i} the word is ${word} 

Script.sh file:

 #!/bin/sh #Set variables i=1 word="dog" #Read in template one line at the time, and replace variables (more #natural (and efficient) way, thanks to Jonathan Leffler). while read line do eval echo "$line" done < "./template.txt" 

Output:

 #sh script.sh the number is 1 the word is dog 

Answer 4

I thought about it again, given the recent interest, and I think the tool that I originally thought about was m4 , the macroprocessor for autotools. Therefore, instead of the variable that I originally indicated, you should use:

 $echo 'I am a DBNAME' | m4 -DDBNAME="database name" 

Answer 5

template.txt

 Variable 1 value: ${var1} Variable 2 value: ${var2} 

data.sh

 #!/usr/bin/env bash declare var1="value 1" declare var2="value 2" 

parser.sh

 #!/usr/bin/env bash # args declare file_data=$1 declare file_input=$2 declare file_output=$3 source $file_data eval "echo \"$(< $file_input)\"" > $file_output 

./parser.sh data.sh template.txt parsed_file.txt

parsed_file.txt

 Variable 1 value: value 1 Variable 2 value: value 2 

Answer 6

here my solution with perl based on the previous answer replaces environment variables:

 perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"missing variable $1")/eg' < infile > outfile 

Answer 7

If you are open to using Perl , this will be my suggestion. Although there are probably some sed and / or AWKs that probably know how to do this a lot easier. If you have a more complicated mapping to more than just dbName for your replacements, you could extend this pretty easily, but you could also add it to the standard Perl script at this point.

 perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql 

A short Perl script makes something a little more complicated (handle multiple keys):

 #!/usr/bin/env perl my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' ); undef $/; my $buf = <STDIN>; $buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace; print $buf; 

If you named the above script as replace-script, you could use it like this:

 replace-script < yourfile | mysql 

Answer 8

Here's a robust Bash function that, despite using eval , should be safe.

All references to ${varName} variables in the input text are extended based on the variables of the calling shell.

Nothing else is revealed: neither references to variables whose names are not enclosed in {...} (for example, $varName ), nor command substitutions ( $(...) and the outdated `...` syntax), nor arithmetic substitutions ( $((...)) and the deprecated syntax $[...] ).

To treat a $ as a literal, \ -escape it; eg:. \${HOME}

Please note that input is only accepted through stdin.

Example:

 $ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded $HOME is "/Users/jdoe"; `date` and $(ls) 

Function Source Code:

 expandVarsStrict(){ local line lineEscaped while IFS= read -r line || [[ -n $line ]]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n # Escape ALL chars. that could trigger an expansion.. IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4') # ... then selectively reenable ${ references lineEscaped=${lineEscaped//$'\4'{/\${} # Finally, escape embedded double quotes to preserve them. lineEscaped=${lineEscaped//\"/\\\"} eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$' done } 

^{The function assumes that there are control characters 0x1 , 0x2 , 0x3 and 0x4 in the input element, because these characters. used internally - since the function processes the text, this should be a safe guess.}

Answer 9

Create rendertemplate.sh :

 #!/usr/bin/env bash eval "echo \"$(cat $1)\"" 

And template.tmpl :

 Hello, ${WORLD} Goodbye, ${CHEESE} 

Highlight the template:

 $ export WORLD=Foo $ CHEESE=Bar ./rendertemplate.sh template.tmpl Hello, Foo Goodbye, Bar 

Answer 10

file.tpl:

 The following bash function should only replace ${var1} syntax and ignore other shell special chars such as `backticks` or $var2 or "double quotes". If I have missed anything - let me know. 

script.sh:

 template(){ # usage: template file.tpl while read -r line ; do line=${line//\"/\\\"} line=${line//\`/\\\`} line=${line//\$/\\\$} line=${line//\\\${/\${} eval "echo \"$line\""; done < ${1} } var1="*replaced*" var2="*not replaced*" template file.tpl > result.txt 

Answer 11

I would suggest using something like Sigil : https://github.com/gliderlabs/sigil

It is compiled into a single binary file, so it is very easy to install on systems.

Then you can make a simple single-line layer as follows:

 cat my-file.conf.template | sigil -p $(env) > my-file.conf 

This is much safer than eval , and simpler using regex or sed

Answer 12

I found this thread, wondering the same thing. It inspired me to do this (careful with back windows)

 $ echo $MYTEST pass! $ cat FILE hello $MYTEST world $ eval echo `cat FILE` hello pass! world 

Answer 13

There are many options, but I realized that I would drop my pile. This is perl based, only the target variables of the form $ {...}, processes the file as an argument and displays the converted file on stdout:

 use Env; Env::import(); while(<>) { $_ =~ s/(\${\w+})/$1/eeg; $text .= $_; } print "$text"; 

Of course, I'm not really a perl person, so it can easily be a fatal flaw (works for me, though).

Answer 14

This can be done in bash if you control the format of the configuration file. You just need to specify (".") The configuration file, and not fine-tune it. This ensures that the variables are created in the context of the current shell (and continue to exist), and not the subshell (where the variable disappears when the subshell exits).

 $ cat config.data export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA export parm_user=pax export parm_pwd=never_you_mind $ cat go.bash . config.data echo "JDBC string is " $parm_jdbc echo "Username is " $parm_user echo "Password is " $parm_pwd $ bash go.bash JDBC string is jdbc:db2://box7.co.uk:5000/INSTA Username is pax Password is never_you_mind 

If your configuration file cannot be a shell script, you can simply “compile” it before execution (compilation depends on your input format).

 $ cat config.data parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL parm_user=pax # user name parm_pwd=never_you_mind # password $ cat go.bash cat config.data | sed 's/#.*$//' | sed 's/[ \t]*$//' | sed 's/^[ \t]*//' | grep -v '^$' | sed 's/^/export ' >config.data-compiled . config.data-compiled echo "JDBC string is " $parm_jdbc echo "Username is " $parm_user echo "Password is " $parm_pwd $ bash go.bash JDBC string is jdbc:db2://box7.co.uk:5000/INSTA Username is pax Password is never_you_mind 

In your specific case, you can use something like:

 $ cat config.data export p_p1=val1 export p_p2=val2 $ cat go.bash . ./config.data echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1" $ bash go.bash select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1 

Then connect the output of go.bash to MySQL and voila, I hope you do not destroy your database :-).