This is because the action systemreturns the completion status of the executed command, which cannot be converted to Int. You must use readProcessto output commands.
> readProcess "date" [] []
"Thu Feb 7 10:03:39 PST 2008\n"
Note that it readProcessdoes not pass the command to the system shell: it runs it directly. The second parameter is the arguments to the command. Therefore your example should be
readProcess "awk" [" { if ( length > x ) { x = length } }END{ print x }", "/home/basic/Desktop/li11112mp/textv"] ""