How to write a regular expression for a unicode name in Java?

I need to write a regex so that I can replace invalid characters in user input before sending it further. I think I need to use string.replaceAll("regex", "replacement") for this. A particular line of code should replace all characters that are not Unicode letters. So this is a whitelist of Unicode characters. Basically, this is checking and replacing invalid username characters.

What I have found so far is this: \p{L}\p{M} , but I'm not sure how to run it in regexp for it to work, as I explained above. Will this be a case of negative expression?

+4
source share
2 answers

Yes, you need denial. The regular expression will be [^\p{L}] for everything except the letters . Another way to write this would be \P{L} .

\p{M} means "all marks", so [^\p{L}\p{M}] means ** everything that is neither a letter, nor a mark. It can also be written as [\P{L}&&[\P{M}]] , but it is not better.

In Java-String, all \ must be doubled, so you write string.replaceAll("[^\\p{L}\\p{M}]", "replacement") .


From the comment:

By the way, regarding your answer, what is included in the category of brands? Do I need this too? Will there be more than just letters for the first name?

This category consists of subcategories.

  • Mn: Mark, Non-Spacing

    An example for this is Μ€ U + 0300. This is COMBINING GRAVE ACCENT, and it can be used together with the letter (letter before) to create accented characters. For commonly used accented characters, there is already a pre-selected form (e.g. Γ© ), but not for others.

  • Mc: Mark, Spacing Combining.

    This is quite rare ... I found them mainly in South Asian scripts and for musical notes. For example, we have U + 1D165, MEN'S SYMBOL COMBINED STEM. ν…¦, which can be combined with U + 1D15D, MUSICAL SYMBOL WHOLE NOTE, 텝, with something like 텝텦. (Hm, the images do not look here. I believe that my browser does not support these characters. Look at the charts if they are not here.)

  • Me: Mark, Enclosing

    These are labels that somehow enclose the base letter (the previous one, if I understand correctly). For example, U + 20DD, ⃝, which allows you to create things like A⃝ . (This should display as A enclosed in a circle, if I understand correctly. This is not the case in my browser). Another will be U + 20E3, ⃣, a COMBINATION KEY, which should display a key cap with a letter on it (A⃣). (They do not appear in my browser. Look at the chart if you do not see them.)

You can find all of them by searching Unicode-Data.txt for ;Mn; ;Mc; or ;Me; , respectively. For more information, see Frequently asked questions: symbols and character combinations .

Do you need them? I'm not sure here. It seems that the most common names (at least in Latin alphabets) will use pre-composed letters. But the user can enter them in a decomposed form - I think that in Mac OS X this is actually the default value. Before filtering unknown characters, you will have to run a normalization algorithm. (Performing normalization seems like a good idea anyway if you want to compare names and not only display them on the screen.)

Edit: not directly related to the question, but related to the discussion in the comments:

I wrote a quick test program to show that [^\pL\pM] not equivalent to [\PL\PM] :

 package de.fencing_game.paul.examples; import java.util.regex.*; public class RegexSample { static String[] regexps = { "[^\\pL\\pM]", "[\\PL\\PM]", ".", "\\pL", "\\pM", "\\PL", "\\PM" }; static String[] strings = { "x", "A", "3", "\n", ".", "\t", "\r", "\f", " ", "-", "!", "Β»", "β€Ί", "β€Ή", "Β«", "Ν³", "Θ", "Ξ£", "Οͺ", "", "Ψ€", "ΰΌ¬", "ΰΌΊ", "ΰΌΌ", "ང", "⃓", "βœ„", "βŸͺ", "γ‚„", "γ‚™", "+", "β†’", "βˆ‘", "−", "β€»", "⁉", "β§“", "β§»", "β‘ͺ", "β’„", "β’°", "β“›", "β“Ά", "\u0300" /* COMBINING GRAVE ACCENT, Mn */, "\u0BCD" /* TAMIL SIGN VIRAMA, Me */, "\u20DD" /* COMBINING ENCLOSING CIRCLE, Me */, "\u2166" /* ROMAN NUMERAL SEVEN, Nl */, }; public static void main(String[] params) { Pattern[] patterns = new Pattern[regexps.length]; System.out.print(" "); for(int i = 0; i < regexps.length; i++) { patterns[i] = Pattern.compile(regexps[i]); System.out.print("| " + patterns[i] + " "); } System.out.println(); System.out.print("-------"); for(int i = 0; i < regexps.length; i++) { System.out.print("|-" + "--------------".substring(0, regexps[i].length()) + "-"); } System.out.println(); for(int j = 0; j < strings.length; j++) { System.out.printf("U+%04x ", (int)strings[j].charAt(0)); for(int i = 0; i < regexps.length; i++) { boolean match = patterns[i].matcher(strings[j]).matches(); System.out.print("| " + (match ? "βœ”" : "-") + " ".substring(0, regexps[i].length())); } System.out.println(); } } } 

Here is the result (from OpenJDK 1.6.0_20 on OpenSUSE):

  | [^\pL\pM] | [\PL\PM] | . | \pL | \pM | \PL | \PM -------|-----------|----------|---|-----|-----|-----|----- U+0078 | - | βœ” | βœ” | βœ” | - | - | βœ” U+0041 | - | βœ” | βœ” | βœ” | - | - | βœ” U+0033 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+000a | βœ” | βœ” | - | - | - | βœ” | βœ” U+002e | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0009 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+000d | βœ” | βœ” | - | - | - | βœ” | βœ” U+000c | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0020 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+002d | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0021 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+00bb | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+203a | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2039 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+00ab | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0373 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0398 | - | βœ” | βœ” | βœ” | - | - | βœ” U+03a3 | - | βœ” | βœ” | βœ” | - | - | βœ” U+03ea | - | βœ” | βœ” | βœ” | - | - | βœ” U+0416 | - | βœ” | βœ” | βœ” | - | - | βœ” U+0624 | - | βœ” | βœ” | βœ” | - | - | βœ” U+0f2c | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0f3a | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0f3c | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0f44 | - | βœ” | βœ” | βœ” | - | - | βœ” U+20d3 | - | βœ” | βœ” | - | βœ” | βœ” | - U+2704 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+27ea | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+3084 | - | βœ” | βœ” | βœ” | - | - | βœ” U+3099 | - | βœ” | βœ” | - | βœ” | βœ” | - U+002b | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2192 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2211 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2222 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+203b | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2049 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+29d3 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+29fb | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+246a | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+2484 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+24b0 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+24db | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+24f6 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” U+0300 | - | βœ” | βœ” | - | βœ” | βœ” | - U+0bcd | - | βœ” | βœ” | - | βœ” | βœ” | - U+20dd | - | βœ” | βœ” | - | βœ” | βœ” | - U+2166 | βœ” | βœ” | βœ” | - | - | βœ” | βœ” 

We can see that:

  • [^\pL\pM] not equivalent [\PL\PM]
  • [\PL\PM] really matches everyone, but
  • still [\PL\PM] not equal . since . does not match \n and \r .

The second point is because [\PL\PM] is union of \PL and \PM : \PL contains characters from all categories other than L (including M), and \PM contains characters from all categories other than M ( including L) - together they contain the entire repertoire of the character.

[^pL\pM] , on the other hand, is a complement to the union of \PL and \PM , which is equivalent to the intersection of \PL and PM .

+7
source

I do not believe that the default Javas regex library (read: outside the ICU link, which I suggest doing even if it requires JNI) supports the Unicode properties you need for this.

If this happened, you would include \p{Diacritic} in your template. But for this you need full support.

I suppose you could shoot for (\pL\pM*)+ , but that doesn’t work for different diacritics: what if someone’s name is not just Γ‰toile , but L'Γ©toile ?

In addition, I thought that the problem of checking people's names was considered almost insoluble, and therefore you should just let people use what they like, maybe the RFC 3454s "stringprep" algorithm has been cleared.

+2
source

All Articles