Are there personalized collections for all international stop-lights?

I am trying to parse utf-8 strings into bite-sized segments. For example, I would like to break the text into "sentences".

Is there a complete collection of characters (or regular expressions) matching the end of sentences in all languages? I am looking for something that will capture the Latin period, exclamation points and interrogations, a Chinese and Japanese full stop, etc.

Something like the above, but for the equivalent of a comma would be great.

+7
source share
3 answers

I have not seen any compilation of such information, and I would expect it to be a serious attempt to collect it. For some commonly used languages, you can get information from The Chicago Manual of Style. There is some information about punctuation marks commonly used in different languages ​​at http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters-other.html , but applies only to a small set of languages ​​and does not distinguish between expressions that terminate characters.

The use of just characters will not be sufficient, since, for example, in English, the full stop is "." occurs in many contexts where he does not interrupt the sentence, as in "for example", or in "1.5".

+3
source

You need to look at the codes with the parameters \p{Sentence_Break=STerm} or \p{Sentence_Break=ATerm} , which also have the property \p{Terminal_Punctuation} . By executing a unichars script against Unicode v6.1, we find out that these code points meet all of these criteria:

 $ unichars -gas '[\p{Sentence_Break=STerm}\p{Sentence_Break=ATerm}]' '\p{Terminal_Punctuation}' U+00021 ‭ ! GC=Po SC=Common EXCLAMATION MARK U+0002E ‭ . GC=Po SC=Common FULL STOP U+0003F ‭ ? GC=Po SC=Common QUESTION MARK U+00589 ‭ ։ GC=Po SC=Common ARMENIAN FULL STOP U+0061F ‭ ؟ GC=Po SC=Common ARABIC QUESTION MARK U+006D4 ‭ ۔ GC=Po SC=Arabic ARABIC FULL STOP U+00700 ‭ ܀ GC=Po SC=Syriac SYRIAC END OF PARAGRAPH U+00701 ‭ ܁ GC=Po SC=Syriac SYRIAC SUPRALINEAR FULL STOP U+00702 ‭ ܂ GC=Po SC=Syriac SYRIAC SUBLINEAR FULL STOP U+007F9 ‭ ߹ GC=Po SC=Nko NKO EXCLAMATION MARK U+00964 ‭ । GC=Po SC=Common DEVANAGARI DANDA U+00965 ‭ ॥ GC=Po SC=Common DEVANAGARI DOUBLE DANDA U+0104A ‭ ၊ GC=Po SC=Myanmar MYANMAR SIGN LITTLE SECTION U+0104B ‭ ။ GC=Po SC=Myanmar MYANMAR SIGN SECTION U+01362 ‭ ። GC=Po SC=Ethiopic ETHIOPIC FULL STOP U+01367 ‭ ፧ GC=Po SC=Ethiopic ETHIOPIC QUESTION MARK U+01368 ‭ ፨ GC=Po SC=Ethiopic ETHIOPIC PARAGRAPH SEPARATOR U+0166E ‭ ᙮ GC=Po SC=Canadian_Aboriginal CANADIAN SYLLABICS FULL STOP U+01803 ‭ ᠃ GC=Po SC=Common MONGOLIAN FULL STOP U+01809 ‭ ᠉ GC=Po SC=Mongolian MONGOLIAN MANCHU FULL STOP U+01944 ‭ ᥄ GC=Po SC=Limbu LIMBU EXCLAMATION MARK U+01945 ‭ ᥅ GC=Po SC=Limbu LIMBU QUESTION MARK U+01AA8 ‭ ᪨ GC=Po SC=Tai_Tham TAI THAM SIGN KAAN U+01AA9 ‭ ᪩ GC=Po SC=Tai_Tham TAI THAM SIGN KAANKUU U+01AAA ‭ ᪪ GC=Po SC=Tai_Tham TAI THAM SIGN SATKAAN U+01AAB ‭ ᪫ GC=Po SC=Tai_Tham TAI THAM SIGN SATKAANKUU U+01B5A ‭ ᭚ GC=Po SC=Balinese BALINESE PANTI U+01B5B ‭ ᭛ GC=Po SC=Balinese BALINESE PAMADA U+01B5E ‭ ᭞ GC=Po SC=Balinese BALINESE CARIK SIKI U+01B5F ‭ ᭟ GC=Po SC=Balinese BALINESE CARIK PAREREN U+01C3B ‭ ᰻ GC=Po SC=Lepcha LEPCHA PUNCTUATION TA-ROL U+01C3C ‭ ᰼ GC=Po SC=Lepcha LEPCHA PUNCTUATION NYET THYOOM TA-ROL U+01C7E ‭ ᱾ GC=Po SC=Ol_Chiki OL CHIKI PUNCTUATION MUCAAD U+01C7F ‭ ᱿ GC=Po SC=Ol_Chiki OL CHIKI PUNCTUATION DOUBLE MUCAAD U+0203C ‭ ‼ GC=Po SC=Common DOUBLE EXCLAMATION MARK U+0203D ‭ ‽ GC=Po SC=Common INTERROBANG U+02047 ‭ ⁇ GC=Po SC=Common DOUBLE QUESTION MARK U+02048 ‭ ⁈ GC=Po SC=Common QUESTION EXCLAMATION MARK U+02049 ‭ ⁉ GC=Po SC=Common EXCLAMATION QUESTION MARK U+02E2E ‭ ⸮ GC=Po SC=Common REVERSED QUESTION MARK U+03002 ‭ 。 GC=Po SC=Common IDEOGRAPHIC FULL STOP U+0A4FF ‭ ꓿ GC=Po SC=Lisu LISU PUNCTUATION FULL STOP U+0A60E ‭ ꘎ GC=Po SC=Vai VAI FULL STOP U+0A60F ‭ ꘏ GC=Po SC=Vai VAI QUESTION MARK U+0A6F3 ‭ ꛳ GC=Po SC=Bamum BAMUM FULL STOP U+0A6F7 ‭ ꛷ GC=Po SC=Bamum BAMUM QUESTION MARK U+0A876 ‭ ꡶ GC=Po SC=Phags_Pa PHAGS-PA MARK SHAD U+0A877 ‭ ꡷ GC=Po SC=Phags_Pa PHAGS-PA MARK DOUBLE SHAD U+0A8CE ‭ ꣎ GC=Po SC=Saurashtra SAURASHTRA DANDA U+0A8CF ‭ ꣏ GC=Po SC=Saurashtra SAURASHTRA DOUBLE DANDA U+0A92F ‭ ꤯ GC=Po SC=Kayah_Li KAYAH LI SIGN SHYA U+0A9C8 ‭ ꧈ GC=Po SC=Javanese JAVANESE PADA LINGSA U+0A9C9 ‭ ꧉ GC=Po SC=Javanese JAVANESE PADA LUNGSI U+0AA5D ‭ ꩝ GC=Po SC=Cham CHAM PUNCTUATION DANDA U+0AA5E ‭ ꩞ GC=Po SC=Cham CHAM PUNCTUATION DOUBLE DANDA U+0AA5F ‭ ꩟ GC=Po SC=Cham CHAM PUNCTUATION TRIPLE DANDA U+0AAF0 ‭ ꫰ GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHAN U+0AAF1 ‭ ꫱ GC=Po SC=Meetei_Mayek MEETEI MAYEK AHANG KHUDAM U+0ABEB ‭ ꯫ GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHEI U+0FE52 ‭ ﹒ GC=Po SC=Common SMALL FULL STOP U+0FE56 ‭ ﹖ GC=Po SC=Common SMALL QUESTION MARK U+0FE57 ‭ ﹗ GC=Po SC=Common SMALL EXCLAMATION MARK U+0FF01 ‭ ! GC=Po SC=Common FULLWIDTH EXCLAMATION MARK U+0FF0E ‭ . GC=Po SC=Common FULLWIDTH FULL STOP U+0FF1F ‭ ? GC=Po SC=Common FULLWIDTH QUESTION MARK U+0FF61 ‭ 。 GC=Po SC=Common HALFWIDTH IDEOGRAPHIC FULL STOP U+11047 ‭ 𑁇 GC=Po SC=Brahmi BRAHMI DANDA U+11048 ‭ 𑁈 GC=Po SC=Brahmi BRAHMI DOUBLE DANDA U+110BE ‭ 𑂾 GC=Po SC=Kaithi KAITHI SECTION MARK U+110BF ‭ 𑂿 GC=Po SC=Kaithi KAITHI DOUBLE SECTION MARK U+110C0 ‭ 𑃀 GC=Po SC=Kaithi KAITHI DANDA U+110C1 ‭ 𑃁 GC=Po SC=Kaithi KAITHI DOUBLE DANDA U+11141 ‭ 𑅁 GC=Po SC=Chakma CHAKMA DANDA U+11142 ‭ 𑅂 GC=Po SC=Chakma CHAKMA DOUBLE DANDA U+11143 ‭ 𑅃 GC=Po SC=Chakma CHAKMA QUESTION MARK U+111C5 ‭ 𑇅 GC=Po SC=Sharada SHARADA DANDA U+111C6 ‭ 𑇆 GC=Po SC=Sharada SHARADA DOUBLE DANDA 

To go the other way around - that is, find the properties specified by the code point, instead of finding the code points based on the set of properties - use the accompanying uniprops script , which retrieves all the properties of the given code point:

 $ uniprops -a . \? \! U+002E ‹.› \N{FULL STOP} \pP \p{Po} All Any ASCII Assigned Basic_Latin Case_Ignorable CI Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct Age=1.1 Block=Basic_Latin Bidi_Class=Common_Separator BC=CS Bidi_Class=CS Block=ASCII BLK=ASCII Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Line_Break=Infix_Numeric LB=IS Line_Break=IS Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0 Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0 IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=AT Sentence_Break=ATerm SB=AT Word_Break=MB Word_Break=MidNumLet WB=MB _Case_Ignorable _X_Begin U+003F ‹?› \N{QUESTION MARK} \pP \p{Po} All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0 Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0 IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST Word_Break=Other WB=XX Word_Break=XX _X_Begin U+0021 ‹!› \N{EXCLAMATION MARK} \pP \p{Po} All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0 Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0 IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST Word_Break=Other WB=XX Word_Break=XX _X_Begin 

I suspect you should check all break-break properties more.

Theres also a 3 rd script in a package, uninames , which does the following:

 $ uninames sentence ; 037E GREEK QUESTION MARK = erotimatiko * sentence-final punctuation * 003B is the preferred character x (question mark - 003F) : 003B semicolon ⁚ 205A TWO DOT PUNCTUATION * historically used to indicate the end of a sentence or change of speaker * extends from baseline to cap height x (presentation form for vertical two dot leader - FE30) x (greek acrophonic epidaurean two - 1015B) 𑂾 110BE KAITHI SECTION MARK * marks end of sentence 

I find these three programs indispensable for exploring Unicode properties. You can install them all with CPAN Unicode :: Tussle suite or test them separately.

+6
source

Chinese, Japanese, and Korean use。. Thais use space. See this list of Unicode full stop equivalents .

+3
source

All Articles