Every byte >= 0x80 is accepted, so UTF-8 identifiers work by accident;
the manual says as much. There is no encoding validation, no
normalisation requirement and no UAX #31 conformance. As a result
${"\xFF\xFE"} is a valid variable name, and $x and $x<U+00A0> are two
distinct variables that render identically.
I am not proposing that PHP accept more characters -- it already accepts
everything. I am proposing a per-file declare under which the accepted
set is specified: well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied. Identifiers consisting only of bytes below 0x80 are never
examined, so existing code pays nothing.
To find out what this would break I surveyed the 250 most-downloaded
Packagist packages and 250 GitHub repositories -- 168,604 PHP files --
using ext/tokenizer. The Packagist corpus contains exactly one
non-ASCII identifier, and no identifier in either corpus is non-NFC.
Tooling, raw data and per-identifier CSVs are here: GitHub - Otzie2023/PHP · GitHub
The patch would not touch the re2c scanner: the byte rule is already
maximally permissive and cannot split a UTF-8 sequence, so a strict
mode only ever rejects and the check can run after the token is formed.
About 13.7 KiB of generated tables, with no dependency on intl, ICU or
mbstring. I would write and maintain it.
Is this worth an RFC, or is there an objection I should know about
before I spend the time?
On Wed, Aug 26, 2026, at 2:42 PM, otzelot2021@outlook.de wrote:
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.
> Do I understand it correctly that by adding that declare to 168604 you
> would uncover a single risky identifier? Not that convincing...
That number is the cost, not the benefit, and I should have separated the
two more clearly.
The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won't notice anything happened.
On 26 August 2026 22:24:06 BST, Larry Garfield <larry@garfieldtech.com> wrote:
On Wed, Aug 26, 2026, at 2:42 PM, otzelot2021@outlook.de wrote:
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check -- a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.
> Do I understand it correctly that by adding that declare to 168604 you
> would uncover a single risky identifier? Not that convincing...
That number is the cost, not the benefit, and I should have separated the
two more clearly.
The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won't notice anything happened.
--Larry Garfield
Is it important enough to have this memory footprint added to each PHP process though?
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check – a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.
Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing…
That number is the cost, not the benefit, and I should have separated the
two more clearly.
The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won’t notice anything happened.
–Larry Garfield
Fair point, Larry.
Moving to enforce strict UTF-8 identifier rules in PHP 9 would definitely simplify things and eliminate edge cases cleanly. The main goal here was to highlight the current ambiguity and explore whether a transitional path or immediate strictness is preferred. Doing a broader ecosystem check before finalizing the PHP 9 deprecation path makes total sense.
The survey answers a different question: if a maintainer adds the declare
to a file they already have, does it still compile? That is adoption
friction, not breakage. It is also a calibration check – a rule that
rejected a lot of legitimate existing code would be the wrong rule, and I
wanted to know that before proposing it rather than after.
Do I understand it correctly that by adding that declare to 168604 you
would uncover a single risky identifier? Not that convincing…
That number is the cost, not the benefit, and I should have separated the
two more clearly.
The single Packagist finding is what would stop compiling: symfony/cache
declares a class whose entire name is the single byte 0xA9. That is the
whole measured adoption cost across the 250 most-installed packages.
If this is so rarely seen in the wild (something that should be verified with more than 250 packages), why make it an option? Just plan that PHP 9 will enforce UTF-8-or-GTFO rules on identifiers, Symfony updates one oddball class, and we move on with life. 99.99% of developers won’t notice anything happened.
–Larry Garfield
Is it important enough to have this memory footprint added to each PHP process though?
About 13.7 KiB of generated tables
cheers
Derick
Hi Derick,
Good point. The ~13.7 KiB footprint comes from the static lookup tables generated for fast classification.
A couple of aspects regarding how this is handled / can be optimized:
Shared Memory (.rodata): Since these tables are declared as static const, in standard multi-process setups (e.g. PHP-FPM), they reside in read-only memory pages shared across processes rather than allocating per-process heap memory.
Compacting / Range Encoding: We can definitely look into compressing the lookup tables (e.g., using run-length/interval encoding or two-stage lookup tables) to bring the table size well under a few kilobytes if the raw table footprint is a concern.
I’m happy to explore compressing the tables or benchmarking the memory impact across different setups to ensure the footprint remains negligible.
Hi Luca & list, Reading this and seeing you talk about making it a compile time error, raises the question for me of how this will interact with variable variables which don’t comply with the proposed rules - AFAICS those wouldn’t be able to be a compile time error and they also wouldn’t have been found in the scan of Packagist files. I imagine “on the fly” class creation, like when mocking code may also run into issues with this up to a point ? Those are also the things which static analysis of code would not be able to find or flag (if this were left to static analysis). Curious to hear your thoughts on this. Smile, Juliette
Not readability – unambiguity. Three concrete things, no jargon:
Two identifiers that look identical on screen are currently two
different identifiers. $x and $x followed by U+00A0 NO-BREAK SPACE
are separate variables. So are “a” plus a combining diaeresis and the
single character U+00E4; both display as a-umlaut.
Text that is not valid UTF-8 at all is currently a valid identifier.
${“\xFF\xFE”} = 1; compiles.
Nobody can state what a PHP identifier is except by copying the byte
class out of the scanner. The manual does exactly that, and so does
PHP-Parser.
The declare says: in this file, an identifier is well-formed UTF-8, is a
Unicode identifier in the sense of UAX #31 (the Unicode annex that
defines this for programming languages), and is written in one canonical
spelling. Anything else is a compile error.
Am 26.08.2026 um 21:57 schrieb Luca Rodenhäuser <otzelot2021@outlook.de>:
What it does solve, ordered by how much I think each is actually worth:
1. PHP has no definition of an identifier. The only answer to "what is a
valid PHP identifier" is "whatever bytes the scanner accepted", which
is why the manual, PHP-Parser, every IDE and every static analyser
each copy out the same byte class. That is a language-definition gap,
not a bug report.
I'm not sure why you consider a formal definition like
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
not a definition. Personally I find this a simpler definition (and easier to implement in a parser) than something like
UTF-8, UAX31-R1-2 with the standard Default-Ignorable Exclusion Profile
But then again I'm not really using non-ASCII identifiers today.
2. Identifiers that render identically are different identifiers. A
no-break space or a decomposed umlaut inside a name is invisible in
every editor. Real, but rare: 11 instances in 168,604 files.
I understand your point. But I'm not so worried about accidental mixups here. And this is also something an LSP or Linter can help you with if it a real concern for you.
3. Case-insensitive lookup folds ASCII only. Stra<U+00DF>e and
STRA<U+00DF>E are the same class; Stra<U+1E9E>e is not, and Strasse is
not. That rule is coherent only if identifiers are ASCII.
Case-insensitive folding adds another problem: Would you be using IntlChar::FOLD_CASE_DEFAULT or IntlChar::FOLD_CASE_EXCLUDE_SPECIAL_I to fold "I"? Or would you base it on a language setting?
In general I think most people consider the case-folding for identifiers nowadays to be a bug, not a feature, so I would probably rather try to reduce than extend it.
What it does solve, ordered by how much I think each is actually worth:
PHP has no definition of an identifier. The only answer to “what is a
valid PHP identifier” is “whatever bytes the scanner accepted”, which
is why the manual, PHP-Parser, every IDE and every static analyser
each copy out the same byte class. That is a language-definition gap,
not a bug report.
I’m not sure why you consider a formal definition like
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
not a definition. Personally I find this a simpler definition (and easier to implement in a parser) than something like
UTF-8, UAX31-R1-2 with the standard Default-Ignorable Exclusion Profile
But then again I’m not really using non-ASCII identifiers today.
Identifiers that render identically are different identifiers. A
no-break space or a decomposed umlaut inside a name is invisible in
every editor. Real, but rare: 11 instances in 168,604 files.
I understand your point. But I’m not so worried about accidental mixups here. And this is also something an LSP or Linter can help you with if it a real concern for you.
Case-insensitive lookup folds ASCII only. Stra<U+00DF>e and
STRA<U+00DF>E are the same class; Stra<U+1E9E>e is not, and Strasse is
not. That rule is coherent only if identifiers are ASCII.
Case-insensitive folding adds another problem: Would you be using IntlChar::FOLD_CASE_DEFAULT or IntlChar::FOLD_CASE_EXCLUDE_SPECIAL_I to fold “I”? Or would you base it on a language setting?
In general I think most people consider the case-folding for identifiers nowadays to be a bug, not a feature, so I would probably rather try to reduce than extend it.
Regards,
Chris
Hi Chris,
Thanks for the solid points. Let me clarify the perspective behind these:
Lexer simplicity vs. Semantic definition:
[a-zA-Z_\x80-\xff] is indeed trivial for the lexer, but it isn’t an identifier specification in terms of character semantics—it’s essentially “ASCII identifiers plus any high byte”. This was originally a pragmatic way to allow Latin-1 / UTF-8 bytes to pass through unchanged. The problem arises when we consider what an identifier semantically is across tooling, ASTs, and static analyzers versus raw byte streaming.
Invisible characters and Linters:
You’re right that linters/LSPs can catch these, but language specifications usually define identifier boundaries (such as TR31 / UAX #31) precisely so that the baseline definition of a valid symbol doesn’t require third-party tooling to reject canonically confusing or invisible code points.
Case Folding:
I completely agree with your assessment here. Extending ASCII case-folding to full Unicode casing (with all the locale subtleties like the dotted/dotless Turkish I) would be opening Pandora’s box. The argument wasn’t necessarily to expand case-folding to Unicode, but rather to highlight the existing inconsistency: PHP treats identifiers as case-insensitive on the ASCII plane while allowing non-ASCII bytes that are strictly case-sensitive.
If the consensus leans toward treating case-insensitivity as historical baggage, clarifying the identifier grammar and transition paths (especially looking ahead to PHP 9 / UTF-8 requirements) is exactly the right discussion to have.
I would like to gauge reaction before writing an RFC.
PHP’s scanner defines identifiers on bytes rather than code points:
LABEL [a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*
Every byte >= 0x80 is accepted, so UTF-8 identifiers work by accident;
the manual says as much. There is no encoding validation, no
normalisation requirement and no UAX #31 conformance. As a result
${“\xFF\xFE”} is a valid variable name, and $x and $x<U+00A0> are two
distinct variables that render identically.
Hi,
There is some confusion here: A valid name is made up of any string, not just strings that are identifiers.
Example of use of a class (more precisely a class alias) named “” (yes, the empty string): https://3v4l.org/TQL18
Of course, names that are not identifiers cannot appear in constructs that accept an identifier only, such as an extends clause.
(…). I am proposing a per-file declare under which the accepted
set is specified: well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied. (…)
I don’t think that a per-file declare is a reasonable option. I would love if the PHP compiler could complain with a clearer error whenever I type accidentally a non-breaking space. But I wouldn’t add a declare clause at the top of each and every file just for that.
On 26 August 2026 16:15:08 BST, "otzelot2021@outlook.de" <otzelot2021@outlook.de> wrote:
... well-formed UTF-8, UAX31-R1-2 with the standard
Default-Ignorable Exclusion Profile, and NFC required rather than
applied
A couple of people have touched on this, but I don't think it's been directly addressed, so I'll ask it more explicitly: what do these terms mean?
- UAX31-R1-2
- the standard Default-Ignorable Exclusion Profile
- NFC
I think that's important context for this discussion, but it's also relevant to the eventual user experience: what is the summary that goes into the manual and error messages? "Class name doesn't meet UAX31-R1-2" would be about as meaningful to most people as the infamous "Unexpected T_PAAMAYIM_NEKUDOTAYIM".
The other thing that I'm not entirely clear on is how much of this is or should be about *rejecting* names, and how much about *normalising* them - bearing in mind we already perform some normalisation in the form of ASCII case folding.
Perhaps we need to step back and define the *problem statement* more clearly, rather than starting with a *solution* and trying to define its benefits?