How can I effectively use the features of the Alex startup code?

Question

I am starting to learn Alex and I think that I have come to the point that the context context will be useful, but I’m not quite sure how to do this. I am trying to use a lex limited subset of erlang binaries . With the following lexer:

{ module Main (main, Token(..), AlexPosn(..), alexScanTokens, token_posn) where } %wrapper "posn" $digit = 0-9 -- digits $alpha = [a-zA-Z] -- alphabetic characters $dbl_quote = \" tokens :- $white+ ; "," { tok (\ps -> Comma p) } "<<" { tok (\ps -> BinaryOpen p) } ">>" { tok (\ps -> BinaryClose p) } $dbl_quote [^$dbl_quote]* $dbl_quote { tok (\ps -> ErlStr p (init (tail s))) } $digit+ { tok (\ps -> ErlInt p (read s)) } { -- action helpers: tok :: (AlexPosn -> String -> Token) -> AlexPosn -> String -> Token tok fps = fps data Token = Comma AlexPosn | BinaryOpen AlexPosn | BinaryClose AlexPosn | ErlInt AlexPosn Integer | ErlStr AlexPosn String deriving (Eq, Show) token_posn :: Token -> AlexPosn token_posn (Comma p) = p token_posn (BinaryOpen p) = p token_posn (BinaryClose p) = p token_posn (ErlInt p _) = p token_posn (ErlStr p _) = p main :: IO () main = do s <- getContents print (alexScanTokens s) } 

I feel very good. For example,

 > alex so_erlang_lexer.x && ghc --make -o erlexer so_erlang_lexer.hs && echo '<<"100", 1>>' | ./erlexer [1 of 1] Compiling Main ( so_erlang_lexer.hs, so_erlang_lexer.o ) Linking erlexer ... [BinaryOpen (AlexPn 0 1 1),ErlStr (AlexPn 2 1 3) "100",Comma (AlexPn 7 1 8),ErlInt (AlexPn 9 1 10) 1,BinaryClose (AlexPn 10 1 11)] 

I would prefer lexed return to be equivalent to Binary [ErlStr "100", ErlInt 1] , but I could not find a lexer that uses the initial codes that click on my head.

• The GHC lexer link provided here does not use any Alex wrappers.
• Alex's own documentation of her monad and monadUserState covers here leaves me with the uncertainty I have to choose and how I can use it.
• Alex's example for tiger is the most promising, but it is so large that I have problems clearing my ignorance.
• This question uses a monad parser, but does not seem to use its state functions.

Will anyone be so kind as to lead me a little?

Answer 1

I'm not quite sure what you are trying to do with lexers, and am knowledgeable enough to guide you about it (but if all you need is useless filter tokens, alex monadic interface seems redundant), anyway here is a sample code for use AlexUserState to accumulate selected tokens using the monadUserState shell.

 { module Main (main) where } %wrapper "monadUserState" $digit = 0-9 -- digits $alpha = [a-zA-Z] -- alphabetic characters $dbl_quote = \" tokens :- $white+ ; "," { ignoreToken } ">" { ignoreToken } $dbl_quote [^$dbl_quote]* $dbl_quote { pushToken $ ErlStr . init . tail } $digit+ { pushToken $ ErlInt . read } { alexEOF :: Alex () alexEOF = return () -- some useful interaces to the Alex monad (which is naturally an instance of state monad) modifyUserState :: (AlexUserState -> AlexUserState) -> Alex () modifyUserState f = Alex (\s -> let st = alex_ust s in Right (s {alex_ust = f st},())) getUserState :: Alex AlexUserState getUserState = Alex (\s -> Right (s,alex_ust s)) -- Token definition minus position information for simplicity data Token = Comma | BinaryOpen | BinaryClose | ErlInt Integer | ErlStr String deriving (Eq, Show) newtype AlexUserState = Binary [Token] deriving (Eq, Show) alexInitUserState :: AlexUserState alexInitUserState = Binary [] -- action helpers: pushToken :: (String -> Token) -> AlexAction () pushToken tokenizer = \(posn,prevChar,pending,s) len -> modifyUserState (push $ take len s) >> alexMonadScan where -- Here tokens are accumulated in reverse order for efficiency. -- You need a more powerful data structure like Data.Sequence to preserve the order. push :: String -> AlexUserState -> AlexUserState push s (Binary ts) = Binary (tokenizer s : ts) ignoreToken :: AlexAction () ignoreToken _ _ = alexMonadScan runAlexScan :: String -> Either String AlexUserState runAlexScan s = runAlex s $ alexMonadScan >> getUserState main :: IO () main = getContents >>= print . runAlexScan } 

But I guess the main problem is that you don't seem to be familiar enough with the concept and use of monads in Haskell yet. The monadic interface of Alex is actually very natural and typical of state monads, and as soon as you have some coding experience, this is something you can easily guess by simply compiling the generated code. (And in case you guessed that type checking is likely to find the corresponding error.)

For this, since there seem to be a lot of good questions and answers about monads, I just refer to Real World Haskell (whose chapter on profiling was especially useful to me.)

But if you already have some knowledge of category theory, perhaps the fastest way to learn monads is to immerse yourself in some relevant documents directly (and please remember that the monad in category theory is a generalization of actions as in the action of groups that fit naturally into the programming context.) For this, see this list of articles on monads and arrows , which includes introductory articles and tutorials for people with specific technical experience.

By the way, I just started to study Erlang. Could you tell me a little about this? Don't you have static typing? Have you tried cloud-haskell and compared it to Erlang? And in what language do you consider yourself the most productive in the context of distributed programming?