CLI shell script code generation from compiled executable?

Question, topic of discussion

I am very interested in creating the source code for a shell command line script from code written in a more reliable, well-executed and platform-independent compiled language (for example, OCaml). Basically, you should program in a compiled language to perform any interactions with the OS that you want (I would suggest: more complex interactions or those that are not easy to make platform independent), and finally, you will compile it to your own binary executable (preferred) that will generate a shell script that affects the shell that you programmed in the compiled language. [ ADDED ]: using "effects" I want to set environment variables and shell parameters, execute certain non-standard commands (the standard glue script will be processed by the compiled executable file and will be stored outside the generated shell script), etc.

I have not yet found such a solution. It seems relatively easy to implement compared to other features today, for example, compiling OCaml for JavaScript.

  • Are there already (public) implementations of what I am describing?
  • What are other features that are (very) similar to what I am describing, and how do they differ from this? (Compiling a language from language (from compiled to sh) comes to mind, although it seems unnecessarily difficult to implement.)

What i don't mean

  • An alternative shell (e.g., Scsh). The systems you administer may not always allow shells to be selected by the user or by one administrator, and I also hope that this system administration solution is exclusively for others (clients, colleagues and others), as well as people who cannot expect take another shell.
  • An alternative interpreter for which non-interactive shell scripts (for example, ocamlscript) are usually used. Personally, I have no problem avoiding shell scripts for this purpose. I do it this way because shell scripts are usually harder to support (for example, sensitive to certain characters and manipulating mutable things like β€œcommands”) and harder to process at the same level of functionality that popular general-purpose programming languages ​​can offer (for example Compare Bash with Python for that matter). However, there are times when you need your own shell script, for example, the shell profile file that the shell received when it was launched.

Background

Practical applications

Some of you may doubt the practical usefulness of what I am describing. One of the practical applications of this is to determine the shell profile based on various conditions (for example, the system platform / OS on which the profile is searched, which follows from the security policy, the specific shell, the type of shell input / non-input, the interactive / non-interactive shell type) . The advantage over a (well-designed) general shell profile as a shell script will be improved performance (native machine code that can generate compressed / optimized source code instead of writing a script by a person), reliability (type checking, exception handling, compile-time checking, cryptographic signing of the resulting binary executable file), features (less or not depend on the CLI tools of the user environment, without limiting the use of the minimum functions the ionicity covered by the CLI tools of all possible platforms) (in practice, standards such as the Single UNIX Specification mean so much, and many shell profile concepts are ported to non-Unix platforms, such as Windows, using PowerShell).

Implementation Details, Side Issues

  • The programmer should be able to control the degree of versatility of the generated shell script. For example, it may happen that a binary executable file is run every time and produces a suitable shell profile code, or it may just generate a file with a fixed shell script adapted to the circumstances of a single run. In the latter case, the listed advantages - in particular, for reliability (for example, exception handling and dependency on user interface tools) are much more limited. [ADDED]
  • Whether the resulting shell of the script in some form is a universal shell of the script (for example, generating GNU autoconf) or a shell of the script adapted (dynamically or not) to the particular shell is not the main question for me.
  • easy *: It seems to me that this can be implemented, basically having the available functions in the library for the main built-in shells. Such a function simply converts itself and the arguments passed into a semantically appropriate and syntactically correct script shell instruction (as a string).

Thanks for any further thoughts, and especially for specific suggestions!

+7
shell haskell scheme ocaml robustness
source share
1 answer

There are no Haskell libraries for this, but you can implement this using abstract syntax trees. I will create a simple toy example that will create a language-neutral abstract syntax tree and then apply a back-end that converts the tree to an equivalent Bash script.

I use two tricks to model syntax trees in Haskell:

  • Model types Bash expressions using GADT
  • DSL implementation using free monads

The GADT trick is pretty simple, and I use several language extensions to sweeten the syntax:

{-# LANGUAGE GADTs , FlexibleInstances , RebindableSyntax , OverloadedStrings #-} import Data.String import Prelude hiding ((++)) type UniqueID = Integer newtype VStr = VStr UniqueID newtype VInt = VInt UniqueID data Expr a where StrL :: String -> Expr String -- String literal IntL :: Integer -> Expr Integer -- Integer literal StrV :: VStr -> Expr String -- String variable IntV :: VInt -> Expr Integer -- Integer variable Plus :: Expr Integer -> Expr Integer -> Expr Integer Concat :: Expr String -> Expr String -> Expr String Shown :: Expr Integer -> Expr String instance Num (Expr Integer) where fromInteger = IntL (+) = Plus (*) = undefined abs = undefined signum = undefined instance IsString (Expr String) where fromString = StrL (++) :: Expr String -> Expr String -> Expr String (++) = Concat 

This allows us to create a typed Bash expression in our DSL. I just performed a few primitive operations, but you could easily imagine how you could extend it to others.

If we did not use language extensions, we could write expressions like:

 Concat (StrL "Test") (Shown (Plus (IntL 4) (IntL 5))) :: Expr String 

This is normal, but not very sexy. The above code uses RebindableSyntax to override numeric literals, so you can replace (IntL n) with n :

 Concat (StrL "Test") (Shown (Plus 4 5)) :: Expr String 

Similarly, I implement Expr Integer Num , so you can add numeric literals with + :

 Concat (StrL "Test") (Shown (4 + 5)) :: Expr String 

Similarly, I use OverloadedStrings so you can replace all occurrences (StrL str) only str :

 Concat "Test" (Shown (4 + 5)) :: Expr String 

I also override the Prelude (++) operator so that we can concatenate expressions as if they were Haskell strings:

 "Test" ++ Shown (4 + 5) :: Expr String 

Besides Shown from integers to strings, it looks just like Haskell's own code. Well maintained!

Now we need a way to create a convenient DSL, preferably with Monad syntactic sugar. Free monads appear here.

Free monads take a functor that represents one step in the syntax tree and creates a syntax tree from it. As a bonus, it is always a monad for any functor, so you can compile these syntax trees using the do notation.

To demonstrate this, I will add more code to the previous code segment:

 -- This is in addition to the previous code {-# LANGUAGE DeriveFunctor #-} import Control.Monad.Free data ScriptF next = NewInt (Expr Integer) (VInt -> next) | NewStr (Expr String ) (VStr -> next) | SetStr VStr (Expr String ) next | SetInt VInt (Expr Integer) next | Echo (Expr String) next | Exit (Expr Integer) deriving (Functor) type Script = Free ScriptF newInt :: Expr Integer -> Script VInt newInt n = liftF $ NewInt n id newStr :: Expr String -> Script VStr newStr str = liftF $ NewStr str id setStr :: VStr -> Expr String -> Script () setStr v expr = liftF $ SetStr v expr () setInt :: VInt -> Expr Integer -> Script () setInt v expr = liftF $ SetInt v expr () echo :: Expr String -> Script () echo expr = liftF $ Echo expr () exit :: Expr Integer -> Script r exit expr = liftF $ Exit expr 

The ScriptF functor is one step in our DSL. Free essentially creates a ScriptF step ScriptF and defines a monad where we can compile lists of these steps. You can think of liftF as doing one step and creating a list with one action.

Then we can use the do notation to assemble these steps, where the do notation combines these action lists:

 script :: Script r script = do hello <- newStr "Hello, " world <- newStr "World!" setStr hello (StrV hello ++ StrV world) echo ("hello: " ++ StrV hello) echo ("world: " ++ StrV world) x <- newInt 4 y <- newInt 5 exit (IntV x + IntV y) 

This shows how we collect the primitive steps that we just defined. This has all the nice features of monads, including support for monadic combinators like forM_ :

 import Control.Monad script2 :: Script () script2 = forM_ [1..5] $ \i -> do x <- newInt (IntL i) setInt x (IntV x + 5) echo (Shown (IntV x)) 

Please note that our nun Script provides type safety, even if our target language can be untyped. You cannot accidentally use a String literal where it expects an Integer or vice versa. You must explicitly convert between them using type-type conversions such as Shown .

Also note that the Script monad swallows any commands after the exit statement. They are ignored before they even reach the translator. Of course, you can change this behavior by rewriting the Exit constructor to take the next next step.

These abstract syntax trees are clean, which means that we can check and interpret them. We can define several backends, such as the Bash backend, which converts our Script monad to the equivalent of a Bash script:

 bashExpr :: Expr a -> String bashExpr expr = case expr of StrL str -> str IntL int -> show int StrV (VStr nID) -> "${S" <> show nID <> "}" IntV (VInt nID) -> "${I" <> show nID <> "}" Plus expr1 expr2 -> concat ["$((", bashExpr expr1, "+", bashExpr expr2, "))"] Concat expr1 expr2 -> bashExpr expr1 <> bashExpr expr2 Shown expr' -> bashExpr expr' bashBackend :: Script r -> String bashBackend script = go 0 0 script where go nStrs nInts script = case script of Free f -> case f of NewInt ek -> "I" <> show nInts <> "=" <> bashExpr e <> "\n" <> go nStrs (nInts + 1) (k (VInt nInts)) NewStr ek -> "S" <> show nStrs <> "=" <> bashExpr e <> "\n" <> go (nStrs + 1) nInts (k (VStr nStrs)) SetStr (VStr nID) e script' -> "S" <> show nID <> "=" <> bashExpr e <> "\n" <> go nStrs nInts script' SetInt (VInt nID) e script' -> "I" <> show nID <> "=" <> bashExpr e <> "\n" <> go nStrs nInts script' Echo e script' -> "echo " <> bashExpr e <> "\n" <> go nStrs nInts script' Exit e -> "exit " <> bashExpr e <> "\n" Pure _ -> "" 

I defined two interpreters: one for the expression syntax tree and one for the monodal DSL syntax tree. These two interpreters compile any language-independent program into the equivalent Bash program, represented as String. Of course, the choice of presentation is entirely up to you.

This interpreter automatically creates new unique variables each time our Script monad requests a new variable.

Try this interpreter and see if it works:

 >>> putStr $ bashBackend script S0=Hello, S1=World! S0=${S0}${S1} echo hello: ${S0} echo world: ${S1} I0=4 I1=5 exit $((${I0}+${I1})) 

It generates a Bash script that executes an equivalent language-dependent program. Similarly, it perfectly converts script2 :

 >>> putStr $ bashBackend script2 I0=1 I0=$((${I0}+5)) echo ${I0} I1=2 I1=$((${I1}+5)) echo ${I1} I2=3 I2=$((${I2}+5)) echo ${I2} I3=4 I3=$((${I3}+5)) echo ${I3} I4=5 I4=$((${I4}+5)) echo ${I4} 

So this is obviously not exhaustive, but hopefully it gives you some ideas on how you will implement this idiom in Haskell. If you want to know more about using free monads, I recommend you read:

I also added the full code here:

 {-# LANGUAGE GADTs , FlexibleInstances , RebindableSyntax , DeriveFunctor , OverloadedStrings #-} import Control.Monad.Free import Control.Monad import Data.Monoid import Data.String import Prelude hiding ((++)) type UniqueID = Integer newtype VStr = VStr UniqueID newtype VInt = VInt UniqueID data Expr a where StrL :: String -> Expr String -- String literal IntL :: Integer -> Expr Integer -- Integer literal StrV :: VStr -> Expr String -- String variable IntV :: VInt -> Expr Integer -- Integer variable Plus :: Expr Integer -> Expr Integer -> Expr Integer Concat :: Expr String -> Expr String -> Expr String Shown :: Expr Integer -> Expr String instance Num (Expr Integer) where fromInteger = IntL (+) = Plus (*) = undefined abs = undefined signum = undefined instance IsString (Expr String) where fromString = StrL (++) :: Expr String -> Expr String -> Expr String (++) = Concat data ScriptF next = NewInt (Expr Integer) (VInt -> next) | NewStr (Expr String ) (VStr -> next) | SetStr VStr (Expr String ) next | SetInt VInt (Expr Integer) next | Echo (Expr String) next | Exit (Expr Integer) deriving (Functor) type Script = Free ScriptF newInt :: Expr Integer -> Script VInt newInt n = liftF $ NewInt n id newStr :: Expr String -> Script VStr newStr str = liftF $ NewStr str id setStr :: VStr -> Expr String -> Script () setStr v expr = liftF $ SetStr v expr () setInt :: VInt -> Expr Integer -> Script () setInt v expr = liftF $ SetInt v expr () echo :: Expr String -> Script () echo expr = liftF $ Echo expr () exit :: Expr Integer -> Script r exit expr = liftF $ Exit expr script :: Script r script = do hello <- newStr "Hello, " world <- newStr "World!" setStr hello (StrV hello ++ StrV world) echo ("hello: " ++ StrV hello) echo ("world: " ++ StrV world) x <- newInt 4 y <- newInt 5 exit (IntV x + IntV y) script2 :: Script () script2 = forM_ [1..5] $ \i -> do x <- newInt (IntL i) setInt x (IntV x + 5) echo (Shown (IntV x)) bashExpr :: Expr a -> String bashExpr expr = case expr of StrL str -> str IntL int -> show int StrV (VStr nID) -> "${S" <> show nID <> "}" IntV (VInt nID) -> "${I" <> show nID <> "}" Plus expr1 expr2 -> concat ["$((", bashExpr expr1, "+", bashExpr expr2, "))"] Concat expr1 expr2 -> bashExpr expr1 <> bashExpr expr2 Shown expr' -> bashExpr expr' bashBackend :: Script r -> String bashBackend script = go 0 0 script where go nStrs nInts script = case script of Free f -> case f of NewInt ek -> "I" <> show nInts <> "=" <> bashExpr e <> "\n" <> go nStrs (nInts + 1) (k (VInt nInts)) NewStr ek -> "S" <> show nStrs <> "=" <> bashExpr e <> "\n" <> go (nStrs + 1) nInts (k (VStr nStrs)) SetStr (VStr nID) e script' -> "S" <> show nID <> "=" <> bashExpr e <> "\n" <> go nStrs nInts script' SetInt (VInt nID) e script' -> "I" <> show nID <> "=" <> bashExpr e <> "\n" <> go nStrs nInts script' Echo e script' -> "echo " <> bashExpr e <> "\n" <> go nStrs nInts script' Exit e -> "exit " <> bashExpr e <> "\n" Pure _ -> "" 
+12
source share

All Articles