ASCII vs Unicode vs UTF-8: Codes and Bytes

Five layers prevent one overloaded word

When someone says “character,” ask what the system actually counts. A user may mean one visible grapheme; a Unicode discussion may mean one code point; JavaScript string indexing exposes UTF-16 code units; a file contains encoded bytes; a debugger renders those bytes with hex or binary digits.

Counts differ without contradiction: this emoji is one code point, two UTF-16 code units, four UTF-8 bytes, and thirty-two displayed bits. A family emoji joined with zero-width joiners can be one grapheme while containing several code points and many bytes. Choose a length unit before enforcing a limit or slicing text.

ASCII is exactly the 7-bit range 0 through 127

The original ASCII coded set contains 128 values. Codes 0 through 31 and 127 are controls; code 32 is space; codes 33 through 126 are graphic characters including letters, digits, and punctuation. The eighth bit of a storage byte is zero for every standard ASCII value.

“Extended ASCII” is not one universal table. Several legacy 8-bit encodings reuse values 128 through 255 differently, so a bare byte such as E9 does not identify its text without an encoding label. The primary historical specification is RFC 20, the US-ASCII format. Use the complete ASCII Table for the defined 0–127 range only.

Unicode assigns code points; it is not an encoding width

Unicode provides a shared character repertoire and properties. A code point is written U+ followed by hexadecimal digits: Latin capital A is U+0041, Hangul syllable 한 is U+D55C, and grinning face is U+1F600. That notation names an abstract value; it is not a promise about bytes in a file.

Unicode code points range through U+10FFFF, excluding surrogate code points reserved for UTF-16 mechanics. UTF-8, UTF-16, and UTF-32 are distinct ways to encode Unicode values. Saying “Unicode file” without naming an encoding leaves byte interpretation unresolved. The latest Unicode Standard defines the repertoire and conformance rules.

UTF-8 maps each Unicode scalar value to one through four bytes

UTF-8 is variable-width and self-synchronizing. ASCII-range code points use one byte; higher values use leading and continuation-byte patterns across two, three, or four bytes. The encoding is uniquely determined for a valid Unicode scalar value; overlong alternatives, surrogate encodings, and values beyond U+10FFFF are invalid.

The WHATWG Encoding Standard’s UTF-8 definition specifies browser decoding behavior. The Binary Translator deliberately uses strict UTF-8 for complete byte sequences; it does not offer a competing selectable ASCII mode because ASCII’s U+0000–U+007F values already use the same one-byte values in UTF-8.

Why ASCII survives unchanged inside UTF-8

Every code point from U+0000 through U+007F maps to one byte with the same numeric value. U+0041 becomes byte 41, and U+0030 becomes byte 30. Therefore a valid ASCII byte sequence is valid UTF-8 and decodes to the same text.

This compatibility does not turn arbitrary high bytes into characters. Byte E9 alone is invalid UTF-8 because it begins a three-byte sequence but lacks two continuation bytes. The character é is UTF-8 bytes C3 A9. A legacy decoder might map lone E9 differently, which is precisely why the encoding label matters.

Numeric zero and the text character “0” are different data

The mathematical integer zero can be written as binary 0 or, at an explicit eight-bit width, 00000000. The character “0” is Unicode U+0030 and UTF-8 byte 30, displayed in binary as 00110000. Interpreted as an unsigned integer, that byte has value 48.

A numeral parser maps the text “0” to a quantity. A text encoder maps the character to bytes. The Binary Converter handles exact integer values; the Binary Translator handles UTF-8 text bytes. State the contract before deciding what a bit string means.

The same-looking text can have different code points and bytes

The visible é can be represented as precomposed U+00E9 or as U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. They commonly render alike, but their UTF-8 bytes differ:

U+00E9             → C3 A9
U+0065 U+0301      → 65 CC 81

Unicode normalization transforms text into a selected canonical or compatibility form. Normalize only according to an application contract: byte-exact signatures and file checksums must preserve original bytes, while search and identifier comparison may require a defined normalization form. Visual equality alone cannot prove byte equality.

JavaScript indexes UTF-16 code units, not UTF-8 bytes

ECMAScript strings are sequences of 16-bit unsigned integer values. A code point outside the Basic Multilingual Plane is represented by a surrogate pair, so "😀".length is 2 even though iteration by code point can yield one item. Neither count reports UTF-8 byte length. See the ECMAScript String type specification.

Use a text encoder to obtain UTF-8 bytes and a text decoder to validate bytes. Do not split arbitrary UTF-8 byte offsets, UTF-16 code-unit offsets, or grapheme boundaries as if they were interchangeable. Each operation needs the unit its protocol defines.

Strict decoding preserves evidence of invalid input

A decoder can reject malformed UTF-8 or replace errors with U+FFFD. Replacement is useful for resilient display, but it loses information: different invalid byte sequences can produce the same visible replacement character. For diagnostics, validation, signatures, and round trips, report the failing offset and retain the original bytes.

Common failures include an unexpected continuation byte, a missing continuation byte, an overlong form, a surrogate value, and an out-of-range code point. A complete octet is exactly eight bits. Reject incomplete trailing groups rather than silently padding them, because adding zeroes changes the byte sequence.

Hex, binary, and Base64 are not character encodings

Hexadecimal and binary are textual notations for numbers or bytes. The UTF-8 bytes for A can be displayed as hex 41 or binary 01000001; the underlying byte is unchanged. Base64 maps bytes into a restricted text alphabet for transport and likewise does not decide what those bytes mean as characters.

The Hex Converter separates an exact hex integer from a UTF-8 hex-byte sequence. Integer 41 means decimal 65; hex byte 41 decodes to A under UTF-8. For hand conversion between numeric bases, use the binary, decimal, hex, and octal guide. For signed fixed-width integers, use the two’s-complement guide.

A practical text-encoding debugging checklist

1. Preserve the original bytes. Do not re-save suspect text before recording a hex dump.
2. Name the expected encoding. Prefer an explicit protocol label; do not infer from a filename alone.
3. Decode strictly. Record the first invalid byte offset and surrounding bytes.
4. Inspect code points. Distinguish invisible controls, combining marks, and confusable characters.
5. Check normalization separately. Valid UTF-8 can still use a different canonical sequence.
6. Count the correct unit. Choose graphemes, code points, UTF-16 code units, or UTF-8 bytes deliberately.
7. Re-encode and compare. A byte-for-byte round trip tests whether information was preserved.