Why Hashes Differ: UTF-8, Newlines and Unicode

Hash functions process bytes, not visual characters

A SHA-256 implementation receives a sequence of bits. It has no concept of letters, emoji, lines, JSON properties, files, or fonts. Software must first encode text into bytes. The LiveParse SHA-256 Generator uses UTF-8, which matches much modern web and API data, but a legacy program might use a locale code page, UTF-16, Latin-1, or another encoding.

ASCII characters U+0000 through U+007F have the same single-byte values in UTF-8 and ASCII, so simple English samples may hide an encoding mismatch. The difference becomes visible with Korean, accented Latin letters, CJK characters, emoji, smart punctuation, and any other non-ASCII content. UTF-8 uses one to four bytes per Unicode scalar value.

A       → 41
é       → C3 A9
한      → ED 95 9C
😀      → F0 9F 98 80

Character count, UTF-16 code-unit count, grapheme count, and UTF-8 byte count are different measurements. An emoji may look like one user-perceived symbol while containing several code points joined into many bytes. A correct hashing tool should report byte length separately from character length.

LF, CRLF, and CR are different byte sequences

Unix-style LF is byte 0A in UTF-8 and other ASCII-compatible encodings. Windows-style CRLF is the two-byte sequence 0D 0A. Historical CR-only text uses 0D. If a document has 100 line breaks, converting LF to CRLF adds 100 bytes and necessarily changes its digest.

Editors can display all three versions as one line containing “hello.” Git can convert checked-out line endings according to configuration and attributes. Template strings, text areas, HTTP libraries, and clipboard operations may also normalize or preserve line breaks differently. Compare bytes at the boundary where hashing actually occurs.

The invisible trailing newline problem

Command-line echo commonly writes a newline after its arguments. Some implementations accept -n to suppress it, but option and escape handling varies across shells. For a reproducible simple Unix-shell test, printf %s is clearer:

printf %s hello | sha256sum
# hashes exactly five bytes

printf '%s\n' hello | sha256sum
# hashes six bytes, including LF

On macOS, replace sha256sum with shasum -a 256. PowerShell string output and pipelines involve their own text and encoding behavior, so hashing an existing binary file with Get-FileHash is easier to reason about than piping formatted console text. When testing a cross-language vector, create and inspect one exact byte fixture.

Many text editors automatically end files with a newline because POSIX text-file conventions and version-control tooling favor it. A browser text field does not invent that final newline unless it is present in the value. Copying the contents of a file into a text area may therefore omit or retain the ending depending on how it was selected.

Visually equivalent Unicode can have different bytes

Unicode can represent some displayed text in more than one canonically equivalent way. The character “é” may be U+00E9 LATIN SMALL LETTER E WITH ACUTE, or U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Fonts commonly render both similarly, but their UTF-8 bytes differ:

U+00E9          → C3 A9
U+0065 U+0301   → 65 CC 81

Normalization Form C (NFC) generally composes canonical sequences when possible; Normalization Form D (NFD) decomposes them. NFKC and NFKD additionally apply compatibility mappings that can collapse distinctions such as presentation forms. None should be applied silently before hashing. A signature, API, or identifier protocol must state the exact normalization and version assumptions.

Case conversion is also not a universal byte normalization. Locale and Unicode rules make case folding more complex than ASCII lowercasing, and changing case changes bytes even when an application treats names case-insensitively. Hash the canonical identifier form owned by that application, not a homemade approximation.

A UTF-8 byte-order mark changes the hash

UTF-8 does not require a byte-order mark, but some software writes the three-byte prefix EF BB BF at the beginning of a file. Other software may strip it when decoding text. Two files that display identically can therefore differ by three leading bytes and have different checksums.

If the task is file verification, hash the raw file exactly as published, BOM included. If the task is a text protocol, follow that protocol's decoding and serialization rules. Do not remove the BOM from a signed file merely to match an expected value; determine which bytes the signer or checksum publisher actually covered.

Spaces, tabs, non-breaking spaces, and zero-width characters

A regular space is UTF-8 byte 20; a horizontal tab is 09. A non-breaking space U+00A0 encodes as C2 A0, and several Unicode zero-width or directional characters may be invisible. Leading spaces, trailing spaces, indentation, and blank lines all participate in the digest.

Copying from rich text, spreadsheets, PDFs, chat tools, or web pages can replace straight quotes, insert non-breaking spaces, or include directionality markers. Inspect code points or a hexadecimal byte dump when pasted values disagree. “Trim the string” is not a neutral fix: it changes data and can break a protocol that treats whitespace as significant.

Objects do not have one automatic hash

JSON objects are semantic name/value collections, but their textual forms can differ in whitespace, property order, escaping, and number spelling. These two texts can describe equivalent data while producing different hashes:

{"a":1,"b":2}
{
  "b": 2,
  "a": 1
}

If a protocol hashes structured data, it needs a canonical serialization or must hash the exact transported bytes. Parsing and reserializing before verification can change property order, Unicode escapes, negative zero, exponent notation, or whitespace. Digital-signature formats define these rules for a reason; follow the format rather than inventing a local formatter.

The same principle applies to YAML, XML, CSV, multipart bodies, archives, and database records. Semantic equality is an application concept. A general hash only answers whether byte sequences agree.

A byte-level hash mismatch checklist

1. Confirm the algorithm and digest encoding. Do not compare SHA-256 with SHA-512, or hex text with undecoded Base64.
2. Compare byte length. A different length immediately proves the inputs differ.
3. Specify the character encoding. Use UTF-8 only when both systems agree.
4. Reveal line endings and the final newline. Inspect LF, CRLF, and CR explicitly.
5. Check for BOM and invisible characters. Use a hex viewer or code-point inspector.
6. Compare Unicode normalization. Apply a form only if the contract defines it.
7. Freeze serialization. Hash the exact source bytes or one defined canonical representation.
8. Create a minimal test vector. Share input bytes in hex plus the complete expected digest.