Bash Templating: How to create configuration files from templates using Bash?

Question

I am writing a script to automate the creation of configuration files for Apache and PHP for my own web server. I do not want to use graphical interfaces like CPanel or ISPConfig.

I have several Apache and PHP configuration file templates. Bash script is necessary to read the templates, make a change of variables and display the analyzed patterns in a folder. What is the best way to do this? I can think of several ways. Which one is the best, or maybe there are some ways to do this? I want to do it in pure Bash (for example, it's just in PHP)

1) How to replace $ {} placeholders in a text file?

template.txt:

the number is ${i} the word is ${word} 

script.sh:

 #!/bin/sh #set variables i=1 word="dog" #read in template one line at the time, and replace variables #(more natural (and efficient) way, thanks to Jonathan Leffler) while read line do eval echo "$line" done < "./template.txt" 

By the way, how do I redirect the output to an external file here? Do I need to avoid something if the variables contain, say, quotation marks?

2) Using cat and sed to replace each variable with its value:

This file template.txt:

 the number is ${i} the word is ${word} 

Team:

 cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/" 

It seems bad to me to avoid many different characters and with many variables, the line length will be longer.

Can you think of another elegant and safe solution?

Answer 1

You can use this:

 perl -p -i -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < template.txt 

replace all the lines of ${...} with the appropriate environment variables (remember to export them before running this script).

For pure bash, this should work (assuming the variables do not contain the string $ {...}):

 #!/bin/bash while read -r line ; do while [[ "$line" =~ (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) ]] ; do LHS=${BASH_REMATCH[1]} RHS="$(eval echo "\"$LHS\"")" line=${line//$LHS/$RHS} done echo "$line" done 

. A solution that does not hang if RHS refers to some variable that refers by itself:

 #! / bin / bash
 line = "$ (cat; echo -na)"
 end_offset = $ {# line}
 while [["$ {line: 0: $ end_offset}" = ~ (. *) (\ $ \ {([a-zA-Z _] [a-zA-Z_0-9] *) \}) (. * )]];  do
     PRE = "$ {BASH_REMATCH [1]}"
     POST = "$ {BASH_REMATCH [4]} $ {line: $ end_offset: $ {# line}}"
     VARNAME = "$ {BASH_REMATCH [3]}"
     eval 'VARVAL = "$' $ VARNAME '"'
     line = "$ PRE $ VARVAL $ POST"
     end_offset = $ {# PRE}
 done
 echo -n "$ {line: 0: -1}"

WARNING I don't know how to properly handle input using NUL in bash or save the number of trailing lines of a new line. The last option is presented as is, because the shell “loves” the binary input:

• read will interpret the backslash.
• read -r will not interpret the backslash, but will omit the last line anyway if it does not end with a new line.
• "$(…)" will highlight as many trailing newline characters as there is, so I finish … with ; echo -na ; echo -na and use echo -n "${line:0:-1}" : this discards the last character (which is equal to a ) and saves as many trailing newlines as possible, as in the input (including not).

Answer 2

Try envsubst

 FOO=foo BAR=bar export FOO BAR envsubst <<EOF FOO is $FOO BAR is $BAR EOF 

Answer 3

envsubst was new to me. Fantastic.

For writing, using heredoc is a great way to conf a file template.

 STATUS_URI="/hows-it-goin"; MONITOR_IP="10.10.2.15"; cat >/etc/apache2/conf.d/mod_status.conf <<EOF <Location ${STATUS_URI}> SetHandler server-status Order deny,allow Deny from all Allow from ${MONITOR_IP} </Location> EOF 

Answer 4

I agree with the use of sed: this is the best search / replace tool. Here is my approach:

 $ cat template.txt the number is ${i} the dog name is ${name} $ cat replace.sed s/${i}/5/ s/${name}/Fido/ $ sed -f replace.sed template.txt > out.txt $ cat out.txt the number is 5 the dog name is Fido 

Answer 5

I think eval works very well. It handles patterns using strings, spaces and all kinds of bash. If you have full control over the templates themselves, of course:

 $ cat template.txt variable1 = ${variable1} variable2 = $variable2 my-ip = \"$(curl -s ifconfig.me)\" $ echo $variable1 AAA $ echo $variable2 BBB $ eval "echo \"$(<template.txt)\"" 2> /dev/null variable1 = AAA variable2 = BBB my-ip = "11.22.33.44" 

This method should be used with caution, of course, since eval can execute arbitrary code. Doing this as root is almost out of the question. Quotes in the template must be escaped, otherwise they will be eaten by eval .

You can also use documents here if you prefer cat - echo

 $ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null 

@plockc dictated a solution that avoids the bash escaping problem:

 $ eval "cat <<EOF $(<template.txt) EOF " 2> /dev/null 

Edit: Removed part of starting this root using sudo ...

Edit: Added comment on how quotation marks should be escaped, added plockc solution to mix!

Answer 6

I have a bash solution like mogsie but with heredoc instead of herestring so you don't escape double quotes

 eval "cat <<EOF $(<template.txt) EOF " 2> /dev/null 

Answer 7

Edit January 6, 2017

I needed to save double quotes in my configuration file, so double escaping double quotes with sed helps:

 render_template() { eval "echo \"$(sed 's/\"/\\\\"/g' $1)\"" } 

I cannot think of saving new lines, but empty lines are kept between them.

---

Although this is an old topic, IMO I found a more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/

 #!/bin/sh # render a template configuration file # expand variables + preserve formatting render_template() { eval "echo \"$(cat $1)\"" } user="Gregory" render_template /path/to/template.txt > path/to/configuration_file 

All loans from Grégory Pakosz .

Answer 8

A longer but more reliable version of the accepted answer:

 perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt 

This extends all instances of $VAR or ${VAR} to their environment values ​​(or, if they are undefined, an empty string).

It properly avoids backslashes and takes a backslash $ to prevent wildcards (unlike envsubst, which turns out to not do this).

So if your environment:

 FOO=bar BAZ=kenny TARGET=backslashes NOPE=engi 

and your template:

 Two ${TARGET} walk into a \\$FOO. \\\\ \\\$FOO says, "Delete C:\\Windows\\System32, it a virus." $BAZ replies, "\${NOPE}s." 

the result will be:

 Two backslashes walk into a \bar. \\ \$FOO says, "Delete C:\Windows\System32, it a virus." kenny replies, "${NOPE}s." 

If you want to avoid backslashes before $ (you can write "C: \ Windows \ System32" in the template unchanged), use this slightly modified version:

 perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt 

Answer 9

I would make it this way, perhaps less efficient, but easier to read / maintain.

 TEMPLATE='/path/to/template.file' OUTPUT='/path/to/output.file' while read LINE; do echo $LINE | sed 's/VARONE/NEWVALA/g' | sed 's/VARTWO/NEWVALB/g' | sed 's/VARTHR/NEWVALC/g' >> $OUTPUT done < $TEMPLATE 

Answer 10

If you want to use Jinja2 templates, see this project: j2cli .

It supports:

• Templates from JSON, INI, YAML files and input streams
• Environment Variable Templates

Answer 11

Instead of reinventing the wheel, use envsubst. It can be used in almost any scenario, for example, to create configuration files from environment variables in docker containers.

If on a Mac make sure you have a homebrew, then bind it to gettext:

 brew install gettext brew link --force gettext 

./template.cfg

 # We put env variables into placeholders here this_variable_1 = ${SOME_VARIABLE_1} this_variable_2 = ${SOME_VARIABLE_2} 

./.env:

 SOME_VARIABLE_1=value_1 SOME_VARIABLE_2=value_2 

./configure.sh

 #!/bin/bash cat template.cfg | envsubst > whatever.cfg 

Now just use this:

 # make script executable chmod +x ./configure.sh # source your variables . .env # export your variables # In practice you may not have to manually export variables # if your solution depends on tools that utilise .env file # automatically like pipenv etc. export SOME_VARIABLE_1 SOME_VARIABLE_2 # Create your config file ./configure.sh 

Answer 12

Accepting a response from ZyX using pure bash, but with a new matching regex style and indirect parameter replacement, it becomes:

 #!/bin/bash regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}' while read line; do while [[ "$line" =~ $regex ]]; do param="${BASH_REMATCH[1]}" line=${line//${BASH_REMATCH[0]}/${!param}} done echo $line done 

Answer 13

If you use Perl , this is an option, and you are satisfied with the basic extensions of only environment variables (unlike all shell variables), consider Stuart P. Bentley a reliable answer .

This answer aims to provide a bash-one solution that, despite using eval , should be safe to use .

goals :

• Support for link extensions ${name} and $name .
• Prevent all other extensions:
  • command substitution ( $(...) and deprecated `...` syntax)
  • arithmetic substitutions ( $((...)) and the deprecated syntax $[...] ).
• Allow selective suppression of variable expansion by prefix \ ( \${name} ).
• Save special characters. at the input, in particular, " and \ instances.
• Allow input through arguments or via stdin.

Function expandVars() :

 expandVars() { local txtToEval=$* txtToEvalEscaped # If no arguments were passed, process stdin input. (( $# == 0 )) && IFS= read -r -d '' txtToEval # Disable command substitutions and arithmetic expansions to prevent execution # of arbitrary commands. # Note that selectively allowing $((...)) or $[...] to enable arithmetic # expressions is NOT safe, because command substitutions could be embedded in them. # If you fully trust or control the input, you can remove the `tr` calls below IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3') # Pass the string to `eval`, escaping embedded double quotes first. # `printf %s` ensures that the string is printed without interpretation # (after processing by by bash). # The `tr` command reconverts the previously escaped chars. back to their # literal original. eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`([' } 

<strong> Examples:

 $ expandVars '\$HOME="$HOME"; `date` and $(ls)' $HOME="/home/jdoe"; `date` and $(ls) # only $HOME was expanded $ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars $SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded 

• For performance reasons, the function automatically reads the stdin input into memory, but easily adapts the function to a linear approach.
• Also supports non-essential variable extensions, such as ${HOME:0:10} , if they do not contain a built-in command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
  • Such embedded substitutions actually BREAK functions (because all instances of $( and ` blindly escaped).
  • Similarly, invalid variable references, such as the ${HOME (no closing } ) BREAK function.
• Due to the bash handling of double-quoted strings, backslashes are processed as follows:
  • \$name prevents the extension.
  • The only \ not followed by $ is preserved as it is.
  • If you want to represent several neighboring instances of \ , you must double them; eg.:
    • \\ → \ - the same as only \
    • \\\\ → \\
  • The input should not contain the following (rarely used) characters that are used for internal purposes: 0x1 , 0x2 , 0x3 .
• To a large extent, the hypothetical problem is that if bash has to introduce a new extension syntax, this function may not interfere with such extensions - see below a solution that does not use eval .

---

If you are looking for a more restrictive solution that only supports the extensions ${name} - that is, with mandatory curly braces, ignoring the $name links - see my answer .

---

Here is an improved version of bash —one, eval —free solution from the accepted answer :

Improvements:

• Support for link extensions ${name} and $name .
• Support for \ -escaping variable references that should not be extended.
• Unlike the above eval solution
  • minor extensions are ignored
  • invalid link references are ignored (they do not violate the script)

  IFS= read -d '' -r lines # read all input from stdin at once end_offset=${#lines} while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do pre=${BASH_REMATCH[1]} # everything before the var. reference post=${BASH_REMATCH[5]}${lines:end_offset} # everything after # extract the var. name; it in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise [[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]} # Is the var ref. escaped, ie, prefixed with an odd number of backslashes? if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then : # no change to $lines, leave escaped var. ref. untouched else # replace the variable reference with the variable value using indirect expansion lines=${pre}${!varName}${post} fi end_offset=${#pre} done printf %s "$lines" 

Answer 14

Here's another clean bash solution:

• using heredoc like this:
  • complexity does not increase due to additional required syntax
  • may contain bash code
    • which also allows you to indent. See below.
• it does not use eval, therefore:
  • no problem displaying trailing blank lines
  • problems with quotation marks in the template

$ cat code

 #!/bin/bash LISTING=$( ls ) cat_template() { echo "cat << EOT" cat "$1" echo EOT } cat_template template | LISTING="$LISTING" bash 

$ cat template (with trailing newlines and double quotes)

 <html> <head> </head> <body> <p>"directory listing" <pre> $( echo "$LISTING" | sed 's/^/ /' ) <pre> </p> </body> </html> 

Exit

 <html> <head> </head> <body> <p>"directory listing" <pre> code template <pre> </p> </body> </html> 

Answer 15

This page describes the answer with awk

 awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt 

Answer 16

Ideal case for shtpl . (My project, so it is not widely used and is not in the documentation. But here is a solution that it offers in any case. Can you test it.)

Just do:

 $ i=1 word=dog sh -c "$( shtpl template.txt )" 

Result:

 the number is 1 the word is dog 

Good luck.

Answer 17

Here is another solution: generate a bash script with all the variables and contents of the template file, this script will look like this:

 word=dog i=1 cat << EOF the number is ${i} the word is ${word} EOF 

If we feed this script in bash, it will give the desired result:

 the number is 1 the word is dog 

Here's how to generate this script and pass this script to bash:

 ( # Variables echo word=dog echo i=1 # add the template echo "cat << EOF" cat template.txt echo EOF ) | bash 

discussion

• A sub-shell opens in brackets; its purpose is to combine all the results
• Inside subclasses we generate all variable declarations
• Also in the subclass, we generate the cat command using HEREDOC
• Finally, we feed the sub-shell output to bash and get the desired result

• If you want to redirect this output to a file, replace the last line as follows:

   ) | bash > output.txt 

Answer 18

You can also use bashible (which internally uses the evaluation approach described above / below).

There is an example of how to create HTML from several parts:

https://github.com/mig1984/bashible/tree/master/examples/templates

Answer 19

 # Usage: template your_file.conf.template > your_file.conf template() { local IFS line while IFS=$'\n\r' read -r line ; do line=${line//\\/\\\\} # escape backslashes line=${line//\"/\\\"} # escape " line=${line//\`/\\\`} # escape ` line=${line//\$/\\\$} # escape $ line=${line//\\\${/\${} # de-escape ${ - allows variable substitution: ${var} ${var:-default_value} etc # to allow arithmetic expansion or command substitution uncomment one of following lines: # line=${line//\\\$\(/\$\(} # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE # line=${line//\\\$\(\(/\$\(\(} # de-escape $(( - allows $(( 1 + 2 )) eval "echo \"${line}\""; done < "$1" } 

This is a pure bash function that is customizable, used in production and should not be interrupted during input. If it breaks, let me know.

Answer 20

Here's a bash function that preserves spaces:

 # Render a file in bash, ie expand environment variables. Preserves whitespace. function render_file () { while IFS='' read line; do eval echo \""${line}"\" done < "${1}" } 

Answer 21

Here's a change to the perl script based on several other answers:

 perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template 

Features (based on my needs, but easy to change):

• Skips escaped parameter extensions (for example, \ $ {VAR}).
• Supports decomposition of form parameters $ {VAR}, but not $ VAR.
• Replaces $ {VAR} with an empty string if there is no VAR-envar.
• It supports only the characters az, AZ, 0-9 and underscores in the name (excluding digits in the first position).

Answer 22

Take a look at the Python script to substitute simple variables here: https://github.com/jeckep/vsubst

It is very simple to use:

 python subst.py --props secure.properties --src_path ./templates --dst_path ./dist 

Answer 23

For one approach to templates, see my answer here .