The theory of language is connected with the theory of computing. Which one is the more philosophical side in the field of computer science, about which programs are possible or what will ever be possible to write, and which problems it is impossible to write an algorithm for solving.
Regular expression is a way of describing a regular language. A regular language is a language that can be solved by a deterministic finite state machine.
You should read the article on finite machines: http://en.wikipedia.org/wiki/Finite_state_machine
And common languages: http://en.wikipedia.org/wiki/Regular_language
All common languages are free context languages, but there are context languages that are not regular. Context Free Language is the set of all strings accepted by Context Free Grammer or Automation Pushdown, which is a single-stack end state machine: http://en.wikipedia.org/wiki/Pushdown_automaton#PDA_and_Context-free_Languages
There are more complex languages that require a Turing machine (any possible program that you can write to your computer) to decide if there is a line in that language or not.
Language theory is also very related to the P vs NP problem and some other interesting things.
In my third year course, Introduction to Computer Science, in the third year it was pretty good to explain this material: Introduction to Computing Theory. Michael Sipser. But it cost me, like $ 160, to buy it new, and it's not very big. Perhaps you can find a used copy or find a copy in the library or something that might help you.
EDIT:
The limitations of Regular Expressions and higher language classes have been investigated per ton over the past 50 years or so. You may be interested in the pumping lemma for regular languages. This is a means of proving that a particular language is not regular:
http://en.wikipedia.org/wiki/Pumping_lemma_for_regular_languages
If the language is not regular, it can be Context Free, which means it can be described by Context Free Grammer, or it can even be in a higher class of languages, you can prove that it is not Context Free by the context transfer pumping lemma Free languages that are similar to those used for regular expressions.
The language can even be unsolvable, which means that even a Turing machine (can program your computer can work) cannot be programmed to choose whether to accept a string in a language or reject it.
I think that you are most interested in finite machines (both deterministic and deterministic) to find out which languages the Regular Expression can choose, and the pumping lemma to prove which languages are not regular.
Basically, a language is not regular if it needs some memory or the ability to count. The language of matching parentheses is not regular, for example, because the machine must remember whether it opened the bracket to find out if it needs to be closed.
The language of all strings using the letters a and b that contain at least three b is the usual language: abababa
The language of all strings using the letters a and b containing more than b than a is not regular.
It also does not follow that all final languages are regular, for example:
The language of all lines less than 50 characters long using the letters a and b that contain more than b than a is regular because it is finite, we know that it can be described as (b | abb | bab | bba | aabbb | ababb | ...) until all possible combinations are indicated.