Name: proposal-regexp-unicode-property-escapes
Owner: Ecma TC39
Description: Proposal to add Unicode property escapes `\p{?}` and `\P{?}` to regular expressions in ECMAScript.
Created: 2016-06-09 12:08:00.0
Updated: 2018-05-20 19:48:33.0
Pushed: 2018-02-01 06:16:41.0
Homepage: https://tc39.github.io/proposal-regexp-unicode-property-escapes/
Size: 126
Language: HTML
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This proposal is at stage 4 of the TC39 process and is scheduled to be included in ES2018.
The Unicode Standard assigns various properties and property values to every symbol. For example, to get the set of symbols that are used exclusively in the Greek script, search the Unicode database for symbols whose Script
property is set to Greek
.
There currently is no way to access these Unicode character properties natively in ECMAScript regular expressions. This makes it painful for developers to support full Unicode in their regular expressions. They currently have two options, neither of which is ideal:
Use a library such as XRegExp to create the regular expressions at run-time:
t regexGreekSymbol = XRegExp('\\p{Greek}', 'A');
xGreekSymbol.test('?');
true
The downside of this approach is that the XRegExp library is a run-time dependency which may not be ideal for performance-sensitive applications. For usage on the web, there is an additional load-time performance penalty: xregexp-all-min.js.gz
takes up over 35 KB of space after minifying and applying gzip compression. Whenever the Unicode Standard is updated, a new version of XRegExp must be published and end users need to update their XRegExp copy in order to use the latest available data.
Use a library such as Regenerate to generate the regular expression at build time:
t regenerate = require('regenerate');
t codePoints = require('unicode-9.0.0/Script/Greek/code-points.js');
t set = regenerate(codePoints);
toString();
'[\u0370-\u0373\u0375-\u0377\u037A-\u037D\u037F\u0384\u0386\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61\u1D66-\u1D6A\u1DBF\u1F00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FC4\u1FC6-\u1FD3\u1FD6-\u1FDB\u1FDD-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFE\u2126\uAB65]|\uD800[\uDD40-\uDD8E\uDDA0]|\uD834[\uDE00-\uDE45]'
magine there?s more code here to save this pattern to a file.
This approach results in optimal run-time performance, although the generated regular expressions tend to be fairly large in size (which could lead to load-time performance problems on the web). The biggest downside is that it requires a build script, which gets painful as the developer needs more Unicode-aware regular expressions. Whenever the Unicode Standard is updated, the build script must be updated and its results must be deployed in order to use the latest available data.
We propose the addition of Unicode property escapes of the form \p{?}
and \P{?}
. Unicode property escapes are a new type of escape sequence available in regular expressions that have the u
flag set. With this feature, the above regular expression could be written as:
t regexGreekSymbol = /\p{Script=Greek}/u;
xGreekSymbol.test('?');
true
This proposal solves all the abovementioned problems:
Unicode property escapes for non-binary Unicode properties look like this:
\p{UnicodePropertyName=UnicodePropertyValue}
The aliases defined in PropertyAliases.txt
and PropertyValueAliases.txt
may be used instead of the canonical property and value names. The use of an unknown property name or value triggers an early SyntaxError
.
For binary properties, the following syntax is available:
\p{LoneUnicodePropertyNameOrValue}
This syntax may also be used as a shorthand for General_Category
values, e.g. \p{Letter}
instead of \p{General_Category=Letter}
.
\P{?}
is the negated form of \p{?}
.
Implementations must support the list of Unicode properties and their property aliases mentioned in the spec proposal. This includes General_Category
, Script
, Script_Extensions
, and some binary properties (including but not limited to Alphabetic
, Uppercase
, Lowercase
, White_Space
, Noncharacter_Code_Point
, Default_Ignorable_Code_Point
, Any
, ASCII
, Assigned
, ID_Start
, ID_Continue
, Join_Control
, Emoji_Presentation
, Emoji_Modifier
, Emoji_Modifier_Base
, etc.). This is a superset of what UTS18 RL1.2 requires. To ensure interoperability, implementations must not extend Unicode property support to the remaining properties.
In regular expressions without the u
flag, the pattern \p
is an (unnecessary) escape sequence for p
. Patterns of the form \p{Letter}
might already be present in existing regular expressions without the u
flag, and therefore we cannot assign new meaning to such patterns without breaking backwards compatibility.
For this reason, ECMAScript 2015 made unnecessary escape sequences like \p
and \P
throw an exception when the u
flag is set. This enables us to change the meaning of \p{?}
and \P{?}
in regular expressions with the u
flag without breaking backwards compatibility.
UAX44-LM3 specifies the loose matching rules for comparing Unicode property and value aliases.
Ignore case, whitespace, underscores, hyphens, [?]
Loose matching makes \p{lB=Ba}
equivalent to \p{Line_Break=Break_After}
or /\p{___lower C-A-S-E___}/u
equivalent to /\p{Lowercase}/u
. We assert that this feature does not add any value, and in fact harms code readability and maintainability.
Should the need arise, then support for loose matching can always be added later, as part of a separate ECMAScript proposal. If we add it now, however, there is no going back.
is
prefix?UAX44-LM3 specifies the loose matching rules for comparing Unicode property and value aliases, one of which is:
Ignore [?] any initial prefix string
is
.
This rule makes Script=IsGreek
and IsScript=Greek
equivalent to Script=Greek
. We assert that this feature does not add any value, and in fact harms code readability. It introduces ambiguity and increases implementation complexity, since some property values or aliases already start with is
, e.g. Decomposition_Type=Isolated
and Line_Break=IS
which is an alias for Line_Break=Infix_Numeric
.
Compatibility with Unicode property escapes in other languages is not an argument either, since no existing regular expression engine seems to implement the is
prefix exactly as described in UAX44-LM3, and those that partially implement it wildly differ in behavior.
Strictness is preferred over ambiguity.
Should the need arise, then support for the is
prefix can always be added later, as part of a separate ECMAScript proposal. If we add it now, however, there is no going back.
\pL
as a shorthand for \p{L}
?This shorthand doesn?t add any value and as such the added implementation complexity (small as it may be) isn?t worth it. \p{L}
works; there?s no reason to introduce another syntax for it other than compatibility with other languages which is an utopian goal anyhow.
Should the need arise, then support for this shorthand can always be added later, as part of a separate ECMAScript proposal. If we add it now, however, there is no going back.
=
(and not something else) as a separator?The =
in \p{?=?}
aligns with the =
in (?=?)
for positive lookaheads and (?<=?)
for positive lookbehinds. Also, =
is what most regular expression engines use as a separator. See issue #8 for more information.
:
as a separator in addition to =
?Supporting multiple separators doesn?t add any value and as such the added implementation complexity (small as it may be) isn?t worth it. \p{Script_Extensions=Greek}
works; there?s no reason to introduce another syntax for it other than compatibility with other languages which is an utopian goal anyhow.
Should the need arise, then support for the :
separator can always be added later, as part of a separate ECMAScript proposal. If we add it now, however, there is no going back.
\p{ScriptName}
as a shorthand for \p{Script=ScriptName}
?In the majority of use cases, Script_Extensions
should be used over Script
. UTS24 explains this nicely with practical examples. As such, it would make more sense to add a shorthand for Script_Extensions
than for Script
. Doing either would cause confusion, however, since the sets of values for these two properties are identical. For example, it wouldn?t be clear if \p{Old_Persian}
refers to the Script
or Script_Extensions
with that name.
\u{?}
instead of adding \p{?}
and \P{?}
?The main argument in favor of overloading \u{?}
is that it hints that it is Unicode. We assert that this hint is unnecessary, as the required u
flag on the regular expression already indicates Unicode.
The p
in \p{?}
stands for ?property?. Combined with the u
flag, this indicates nicely that the expression within the braces relates to a Unicode property.
Overloading \u{?}
introduces an ambiguity. Imagine a new binary property or general category named Beef
is added to the Unicode Standard. Since Beef
consists of hexadecimal digits only ([A-Fa-f0-9]
), it?s unclear whether \u{Beef}
is a code point escape sequence for U+BEEF HANGUL SYLLABLE BBEGS or whether it?s a property escape sequence referring to the property/category named Beef
.
Existing other languages with support for Unicode property escapes use \p{?}
and \P{?}
. Although compatibility with these other implementations is a non-goal (since they?re not compatible amongst themselves to begin with), it makes sense to follow the tradition here and re-use the base syntax that developers are already familiar with.
Name
property (\p{Name=?}
)?Developers already have a way to refer to a specific symbol without having to use that symbol in their source code: Unicode code point escapes of the form \u{1D306}
. As such, the need to support \p{Name=TETRAGRAM FOR CENTRE}
is not strong enough to warrant inclusion in this proposal.
Support for the Name
property can always be added later, as part of a separate ECMAScript proposal. If we add it now, however, there is no going back.
\d
To match any decimal number in Unicode rather than just ASCII [0-9]
, use \p{Decimal_Number}
instead of \d
as per UTS18.
t regex = /^\p{Decimal_Number}+$/u;
x.test('????????????????');
true
\D
To match any Unicode symbol that is not a decimal number rather than just [^0-9]
, use \P{Decimal_Number}
instead of \D
.
t regex = /^\P{Decimal_Number}+$/u;
x.test('?? ??????? ?? ? ?????????');
true
\w
To match any word symbol in Unicode rather than just ASCII [a-zA-Z0-9_]
, use [\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\p{Join_Control}]
as per UTS18.
t regex = /([\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\p{Join_Control}]+)/gu;
t text = `
ric: ?? ?????? ??? ????? ?????
ali: ???? ??????????? ????? ???-? ??? ??? ????
gian: ???? ??????? ??????? ??????? ?????? ????????????
donian: ????? ??????? ?????? ? ????? ?? ??????
namese: Tàu cánh ng?m c?a tôi ??y l??n
match;
e (match = regex.exec(text)) {
nst word = match[1];
nsole.log(`Matched word with length ${ word.length }: ${ word }`);
Console output:
hed word with length 7: Amharic
hed word with length 2: ??
hed word with length 6: ??????
hed word with length 3: ???
hed word with length 5: ?????
hed word with length 5: ?????
hed word with length 7: Bengali
hed word with length 4: ????
hed word with length 11: ???????????
hed word with length 5: ?????
hed word with length 3: ???
hed word with length 1: ?
hed word with length 3: ???
hed word with length 3: ???
hed word with length 4: ????
hed word with length 8: Georgian
hed word with length 4: ????
hed word with length 7: ???????
hed word with length 7: ???????
hed word with length 7: ???????
hed word with length 6: ??????
hed word with length 12: ????????????
hed word with length 10: Macedonian
hed word with length 5: ?????
hed word with length 7: ???????
hed word with length 6: ??????
hed word with length 1: ?
hed word with length 5: ?????
hed word with length 2: ??
hed word with length 6: ??????
hed word with length 10: Vietnamese
hed word with length 3: Tàu
hed word with length 4: cánh
hed word with length 4: ng?m
hed word with length 3: c?a
hed word with length 3: tôi
hed word with length 3: ??y
hed word with length 4: l??n
\W
To match any non-word symbol in Unicode rather than just [^a-zA-Z0-9_]
, use [^\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\p{Join_Control}]
.
To match emoji symbols, the binary properties from UTR51 come in handy.
t regex = /\p{Emoji_Modifier_Base}\p{Emoji_Modifier}?|\p{Emoji_Presentation}|\p{Emoji}\uFE0F/gu;
This regular expression matches, from left to right:
\p{Emoji_Modifier_Base}\p{Emoji_Modifier}?
);\p{Emoji_Presentation}
);\p{Emoji}\uFE0F
).t regex = /\p{Emoji_Modifier_Base}\p{Emoji_Modifier}?|\p{Emoji_Presentation}|\p{Emoji}\uFE0F/gu;
t text = `
31A}: ? default emoji presentation character (Emoji_Presentation)
194}\u{FE0F}: ?? default text presentation character rendered as emoji
F469}: ? emoji modifier base (Emoji_Modifier_Base)
F469}\u{1F3FF}: ?? emoji modifier base followed by a modifier
match;
e (match = regex.exec(text)) {
nst emoji = match[0];
nsole.log(`Matched sequence ${ emoji } ? code points: ${ [...emoji].length }`);
Console output:
hed sequence ? ? code points: 1
hed sequence ? ? code points: 1
hed sequence ?? ? code points: 2
hed sequence ?? ? code points: 2
hed sequence ? ? code points: 1
hed sequence ? ? code points: 1
hed sequence ?? ? code points: 2
hed sequence ?? ? code points: 2
Match any numeric symbol in Unicode, including non-decimal symbols such as Roman numerals:
t regex = /^\p{Number}+$/u;
x.test('²³¹¼½¾???????????????????????????????????????????????????');
true
Match ECMAScript IdentifierStart
or IdentifierPart
symbols without the need for complex regular expressions generated by build scripts:
t regexIdentifierStart = /[$_\p{ID_Start}]/u;
t regexIdentifierPart = /[$_\u200C\u200D\p{ID_Continue}]/u;
t regexIdentifierName = /^(?:[$_\p{ID_Start}])(?:[$_\u200C\u200D\p{ID_Continue}])*$/u;
{ unicodePropertyEscape: true }
option enabled