Commit 0ddfe63bc4 for perl
commit 0ddfe63bc4daaac94902f81a8af715d247f7f6a2
Author: Yves Orton <demerphq@gmail.com>
Date: Tue Oct 20 15:15:52 2026 +0200
unicode - clarify that internals often use "native-Unicode" codepoints
Until today I didn't properly appreciate that internally our codepoints
are not true Unicode codepoints, but actually hybrid codepoints where
0-255 map to the native encoding, and codepoints 256 and above follow
Unicode. In ASCII environments the native codepoints and Unicode
codepoints are identical, but in EBCDIC they aren't.
So, "A" does not have codepoint 65 on EBCDIC. However this doesn't
matter because most of the functions you might use on "A" which cares
what codepoint is has will do the right thing in EBCDIC with the value
it does have.
Anyway, this patch updates some of the docs to use the term
native-Unicode for the unicode apis - and provides a definition of the
term so someone just reading a macro definition doesn't get confused.
It also explains the subtlties here a bit more clearly. I kind of think
that a function which takes a native-Unicode codepoint should call it
ncp not cp as well.
It does not change any code, just renames a var and update the comments
to help others avoid the confustion I experienced here.
diff --git a/autodoc.pl b/autodoc.pl
index 86fdb6c7d7..38e63d606e 100644
--- a/autodoc.pl
+++ b/autodoc.pl
@@ -3233,6 +3233,21 @@ $api{hdr} = <<"_EOB_";
|the UTF- encodings, and a string encoded in UTF-EBCDIC may occupy a different
|number of bytes than in UTF-8.
|
+|To abstract away platform differences such as EBCDIC, many Perl string APIs
+|use Perl's "native-Unicode" code point numbering. Values from 0 through 255
+|use the platform's native character numbers, and values above 255 use Unicode
+|numbering. For example, the native-Unicode value of C<'A'> is 65 on ASCII
+|platforms and 193 on EBCDIC platforms, while LATIN CAPITAL LETTER A WITH
+|MACRON (U+0100) is 256 on both.
+|
+|C<NATIVE_TO_UNI> and C<UNI_TO_NATIVE> convert between native-Unicode and
+|standard Unicode code point numbers. Most internal Perl code should not
+|need these conversions. They are usually needed when exchanging code points
+|with an external interface or storage format. Some older API names contain
+|C<uni> even though their arguments and results use native-Unicode numbering.
+|Other APIs use standard Unicode numbering. See L<perlguts/Unicode Support>
+|and the individual API descriptions for details.
+|
|The organization of this document is tentative and subject to change.
|Suggestions and patches welcome
|L<perl5-porters\@perl.org|mailto:perl5-porters\@perl.org>.
diff --git a/embed.fnc b/embed.fnc
index f0633e4bb9..2a5b4233ac 100644
--- a/embed.fnc
+++ b/embed.fnc
@@ -4022,7 +4022,7 @@ Cp |UV |to_uni_lower |UV c \
Cp |UV |to_uni_title |UV c \
|NN U8 *p \
|NN STRLEN *lenp
-Cp |UV |to_uni_upper |UV c \
+Cp |UV |to_uni_upper |UV uv \
|NN U8 *p \
|NN STRLEN *lenp
Cp |UV |to_utf8_fold_flags_ \
diff --git a/handy.h b/handy.h
index 7893ddbf50..b939608f3f 100644
--- a/handy.h
+++ b/handy.h
@@ -1248,10 +1248,10 @@ There is no C<toUPPER_L1> nor C<toUPPER_LATIN1> as the uppercase of some code
points in the 0..255 range is above that range or consists of multiple
characters. Instead use C<toUPPER_uvchr>.
-C<toUPPER_uvchr> returns the uppercase of any Unicode code point. The return
-value is identical to that of C<toUPPER_A> for input code points in the ASCII
-range. The uppercase of the vast majority of Unicode code points is the same
-as the code point itself. For these, and for code points above the legal
+C<toUPPER_uvchr> returns the uppercase of any native-Unicode code point. The
+return value is identical to that of C<toUPPER_A> for input code points in the
+ASCII range. The uppercase of the vast majority of Unicode code points is the
+same as the code point itself. For these, and for code points above the legal
Unicode maximum, this returns the input code point unchanged. It additionally
stores the UTF-8 of the result into the buffer beginning at C<s>, and its
length in bytes into C<*lenp>. The caller must have made C<s> large enough to
@@ -1302,15 +1302,20 @@ There is no C<toFOLD_L1> nor C<toFOLD_LATIN1> as the foldcase of some code
points in the 0..255 range is above that range or consists of multiple
characters. Instead use C<toFOLD_uvchr>.
-C<toFOLD_uvchr> returns the foldcase of any Unicode code point. The return
-value is identical to that of C<toFOLD_A> for input code points in the ASCII
-range. The foldcase of the vast majority of Unicode code points is the same
-as the code point itself. For these, and for code points above the legal
+C<toFOLD_uvchr> returns the foldcase of any native-Unicode code point. The
+return value is identical to that of C<toFOLD_A> for input code points in the
+ASCII range. The foldcase of the vast majority of Unicode code points is the
+same as the code point itself. For these, and for code points above the legal
Unicode maximum, this returns the input code point unchanged. It additionally
stores the UTF-8 of the result into the buffer beginning at C<s>, and its
length in bytes into C<*lenp>. The caller must have made C<s> large enough to
contain at least C<UTF8_MAXBYTES_CASE+1> bytes to avoid possible overflow.
+The return value is also a native-Unicode code point. Native-Unicode and
+Unicode code point values are the same on ASCII platforms, but can differ below
+256 on EBCDIC platforms. The older name C<to_uni_fold> uses native-Unicode
+code points as well.
+
NOTE: the foldcase of a code point may be more than one code point. The
return value of this function is only the first of these. The entire foldcase
is returned in C<s>. To determine if the result is more than a single code
@@ -1359,10 +1364,10 @@ C<toLOWER_LC> returns the lowercase of the input code point according to the
rules of the current POSIX locale. Input code points outside the range 0..255
are returned unchanged.
-C<toLOWER_uvchr> returns the lowercase of any Unicode code point. The return
-value is identical to that of C<toLOWER_L1> for input code points in the 0..255
-range. The lowercase of the vast majority of Unicode code points is the same
-as the code point itself. For these, and for code points above the legal
+C<toLOWER_uvchr> returns the lowercase of any native-Unicode code point. The
+return value is identical to that of C<toLOWER_L1> for input code points in the
+0..255 range. The lowercase of the vast majority of Unicode code points is the
+same as the code point itself. For these, and for code points above the legal
Unicode maximum, this returns the input code point unchanged. It additionally
stores the UTF-8 of the result into the buffer beginning at C<s>, and its
length in bytes into C<*lenp>. The caller must have made C<s> large enough to
@@ -1409,10 +1414,10 @@ There is no C<toTITLE_L1> nor C<toTITLE_LATIN1> as the titlecase of some code
points in the 0..255 range is above that range or consists of multiple
characters. Instead use C<toTITLE_uvchr>.
-C<toTITLE_uvchr> returns the titlecase of any Unicode code point. The return
-value is identical to that of C<toTITLE_A> for input code points in the ASCII
-range. The titlecase of the vast majority of Unicode code points is the same
-as the code point itself. For these, and for code points above the legal
+C<toTITLE_uvchr> returns the titlecase of any native-Unicode code point. The
+return value is identical to that of C<toTITLE_A> for input code points in the
+ASCII range. The titlecase of the vast majority of Unicode code points is the
+same as the code point itself. For these, and for code points above the legal
Unicode maximum, this returns the input code point unchanged. It additionally
stores the UTF-8 of the result into the buffer beginning at C<s>, and its
length in bytes into C<*lenp>. The caller must have made C<s> large enough to
diff --git a/intrpvar.h b/intrpvar.h
index 97f79b67d0..24d129ac09 100644
--- a/intrpvar.h
+++ b/intrpvar.h
@@ -1049,6 +1049,9 @@ PERLVARA(I, TR_SPECIAL_HANDLING_UTF8, UTF8_MAXBYTES, char)
PERLVAR(I, AboveLatin1, SV *)
PERLVAR(I, Assigned_invlist, SV *)
PERLVAR(I, GCB_invlist, SV *)
+/* Inversion list of characters whose full case fold has more than one
+ * character. The code points use native-Unicode numbering, as in the other
+ * property lists. */
PERLVAR(I, HasMultiCharFold, SV *)
PERLVAR(I, InMultiCharFold, SV *)
PERLVAR(I, Latin1, SV *)
diff --git a/proto.h b/proto.h
index d60ee183fe..b6ee43aecb 100644
--- a/proto.h
+++ b/proto.h
@@ -9939,7 +9939,7 @@ Perl_to_uni_title(pTHX_ UV c, U8 *p, STRLEN *lenp)
} STMT_END
PERL_CALLCONV UV
-Perl_to_uni_upper(pTHX_ UV c, U8 *p, STRLEN *lenp)
+Perl_to_uni_upper(pTHX_ UV uv, U8 *p, STRLEN *lenp)
Perl_attribute_nonnull_aTHX
Perl_attribute_nonnull(pTHX_2)
Perl_attribute_nonnull(pTHX_3);
diff --git a/utf8.c b/utf8.c
index 2ba48df950..006b8c25ed 100644
--- a/utf8.c
+++ b/utf8.c
@@ -58,6 +58,15 @@ strings. For the uninitiated, this is a method of representing arbitrary
Unicode characters as a variable number of bytes, in such a way that
characters in the ASCII range are unmodified, and a zero byte never appears
within non-zero characters.
+
+On EBCDIC platforms, UTF-8 in this file means UTF-EBCDIC. The code point
+arguments and results generally use Perl's "native-Unicode" numbering:
+values from 0 through 255 use the platform's native character set, and values
+above 255 use Unicode numbering. Thus 'A' is 65 on ASCII platforms and 193 on
+EBCDIC platforms, while U+0100 is 256 on both. UNI_TO_NATIVE and NATIVE_TO_UNI
+convert between standard Unicode and native-Unicode numbers, as described in
+utf8.h. The encoding and decoding functions translate between native-Unicode
+numbers and UTF-8 or UTF-EBCDIC strings.
*/
void
@@ -3688,23 +3697,25 @@ Perl_to_upper_title_latin1_(pTHX_ const U8 c, U8* p, STRLEN *lenp,
"foldcase")
UV
-Perl_to_uni_upper(pTHX_ UV c, U8* p, STRLEN *lenp)
+Perl_to_uni_upper(pTHX_ UV uv, U8* p, STRLEN *lenp)
{
PERL_ARGS_ASSERT_TO_UNI_UPPER;
- /* Convert the Unicode character whose ordinal is <c> to its uppercase
- * version and store that in UTF-8 in <p> and its length in bytes in <lenp>.
- * Note that the <p> needs to be at least UTF8_MAXBYTES_CASE+1 bytes since
- * the changed version may be longer than the original character.
+ /* Convert the character whose native-Unicode code point is <uv> to its
+ * uppercase version and store that in UTF-8 (UTF-EBCDIC on EBCDIC platforms)
+ * in <p> and its length in bytes in <lenp>. Note that the <p> needs to be
+ * at least UTF8_MAXBYTES_CASE+1 bytes since the changed version may be
+ * longer than the original
+ * character.
*
* The ordinal of the first character of the changed version is returned
* (but note, as explained above, that there may be more.) */
- if (c < 256) {
- return to_upper_title_latin1_((U8) c, p, lenp, 'S');
+ if (uv < 256) {
+ return to_upper_title_latin1_((U8) uv, p, lenp, 'S');
}
- return CALL_UPPER_CASE(c, NULL, p, lenp);
+ return CALL_UPPER_CASE(uv, NULL, p, lenp);
}
UV
diff --git a/utf8.h b/utf8.h
index 73117a7cc0..979fb612c5 100644
--- a/utf8.h
+++ b/utf8.h
@@ -19,6 +19,14 @@
* outside some of the macros in this this file, the differences are hopefully
* invisible at the semantic level.
*
+ * "Native-Unicode" is Perl's code point numbering: values from 0 through 255
+ * use the platform's native character set, and values above 255 use Unicode
+ * numbering. For example, the native-Unicode value of 'A' is 65 on ASCII
+ * platforms and 193 on EBCDIC platforms, while U+0100 is 256 on both.
+ * UNI_TO_NATIVE and NATIVE_TO_UNI convert between standard Unicode and
+ * native-Unicode numbers. The encoders and decoders translate between these
+ * numbers and UTF-8, or UTF-EBCDIC on EBCDIC platforms.
+ *
* UTF-EBCDIC has an isomorphic translation named I8 (for "Intermediate eight")
* which differs from UTF-8 only in a few details. It is often useful to
* translate UTF-EBCDIC into this form for processing. In general, macros and
@@ -275,6 +283,9 @@ the implementation.
For conversion of code points potentially larger than will fit in a character,
use L</NATIVE_TO_UNI>.
+The result can be used, for example, to look up a character in a table indexed
+by Latin-1 code point.
+
=for apidoc Am|U8|LATIN1_TO_NATIVE|U8 ch
Returns the native equivalent of the input Latin-1 code point (including ASCII
@@ -287,21 +298,74 @@ the implementation.
For conversion of code points potentially larger than will fit in a character,
use L</UNI_TO_NATIVE>.
+This is useful for character constants written as Latin-1 numbers. For
+example, C<LATIN1_TO_NATIVE(0xDF)> gives the native code point for LATIN SMALL
+LETTER SHARP S. Character constants such as C<'A'> already have their native
+values, so no conversion is needed for them.
+
=for apidoc Am|UV|NATIVE_TO_UNI|UV ch
-Returns the Unicode equivalent of the input native code point given by C<ch>.
-Thus, C<NATIVE_TO_UNI(195)> on EBCDIC platforms returns 67. These each
-represent the character C<"C"> on their respective platforms. On ASCII
-platforms no conversion is needed, so this macro expands to just its input,
-adding no time nor space requirements to the implementation.
+Converts C<ch> from Perl's native-Unicode code point numbering to standard
+Unicode code point numbering. Native-Unicode uses the platform's native
+character numbers for 0..255 and standard Unicode numbers for values above
+255. Thus, C<NATIVE_TO_UNI(195)> on EBCDIC platforms returns 67; both numbers
+represent C<"C"> in their respective numbering conventions. Values above 255
+are returned unchanged. On ASCII platforms no conversion is needed, so this
+macro expands to just its input, adding no time nor space requirements to the
+implementation.
+
+This is useful when comparing a character with a code point number from the
+Unicode Standard, or when passing it to a library that uses Unicode code point
+numbers. For example, C<NATIVE_TO_UNI('A')> is 0x41 on both ASCII and EBCDIC
+platforms, even though the value of C<'A'> differs between them.
+
+It is generally unnecessary between Perl character APIs that already use
+native-Unicode numbering, including L</toUPPER_uvchr> and L</toFOLD_uvchr>.
+Check both APIs' contracts before adding a conversion. This macro converts
+a code point number, not an encoded string. See L</UNI_TO_NATIVE> for the
+reverse conversion.
=for apidoc Am|UV|UNI_TO_NATIVE|UV ch
-Returns the native equivalent of the input Unicode code point given by C<ch>.
-Thus, C<UNI_TO_NATIVE(68)> on EBCDIC platforms returns 196. These each
-represent the character C<"D"> on their respective platforms. On ASCII
-platforms no conversion is needed, so this macro expands to just its input,
-adding no time nor space requirements to the implementation.
+Converts C<ch> from standard Unicode code point numbering to Perl's
+native-Unicode numbering. Native-Unicode uses the platform's native character
+numbers for 0..255 and standard Unicode numbers for values above 255. Thus,
+C<UNI_TO_NATIVE(68)> on EBCDIC platforms returns 196; both numbers represent
+C<"D"> in their respective numbering conventions. Values above 255 are
+returned unchanged. On ASCII platforms no conversion is needed, so this
+macro expands to just its input, adding no time nor space requirements to the
+implementation.
+
+This is useful when a character is specified by its Unicode code point number.
+For example, the Unicode code point for LATIN SMALL LETTER SHARP S is 0xDF.
+The following writes that character in UTF-8 on an ASCII platform, or
+UTF-EBCDIC on an EBCDIC platform:
+
+ U8 encoded[UTF8_MAXBYTES + 1];
+ UV sharp_s = UNI_TO_NATIVE(0xDF);
+ uv_to_utf8(encoded, sharp_s);
+
+Most Perl character-handling functions already use native-Unicode code points.
+For example, C<valid_utf8_to_uv> returns the native-Unicode code point that
+C<uv_to_utf8> expects. Passing that result through C<UNI_TO_NATIVE> would
+treat the native-Unicode value as a standard Unicode number, possibly changing
+the character on EBCDIC platforms. Such a mistake has no effect on ASCII
+platforms, where the native-Unicode and standard Unicode code point values are
+the same.
+
+Some older function names contain C<uni>, such as C<to_uni_fold>, even though
+they too work with native-Unicode code points. Their results can be passed
+directly to other functions that take native-Unicode code points, without
+conversion. For example, to uppercase Unicode U+00DF:
+
+ U8 result[UTF8_MAXBYTES_CASE + 1];
+ STRLEN length;
+ UV first = toUPPER_uvchr(UNI_TO_NATIVE(0xDF), result, &length);
+
+C<first> uses native-Unicode numbering, and C<result> contains the complete
+uppercase mapping in native UTF-8 (UTF-8 on ASCII platforms, UTF-EBCDIC on
+EBCDIC platforms). See L</NATIVE_TO_UNI> for the reverse numbering conversion.
+Neither macro encodes or decodes a string.
=cut
*/
@@ -613,8 +677,8 @@ regen/charset_translations.pl. */
=for apidoc Am|STRLEN|UVCHR_SKIP|UV cp
returns the number of bytes required to represent the code point C<cp> when
-encoded as UTF-8. C<cp> is a native (ASCII or EBCDIC) code point if less than
-255; a Unicode code point otherwise.
+encoded as UTF-8. C<cp> uses native-Unicode numbering: a native (ASCII or
+EBCDIC) code point if less than 256; a Unicode code point otherwise.
=cut
*/