My Favorite Bugs: Invalid Surrogate Pairs

Developers have confirmed a bug in their real-time collaborative editing tool that causes silent data loss when inserting certain emojis, specifically those requiring surrogate pairs in Unicode. This issue impacts the reliability of content synchronization in web-based editors, raising concerns for developers working with Unicode and emoji input.

The bug was discovered during debugging of a migration to a collaborative editor built on TipTap, ProseMirror, and Yjs. Developers observed that when users inserted emojis above U+FFFF—such as 🤠 or 👩‍🚀—the underlying CRDT library would splice surrogate pairs, breaking the string and causing encoding errors. This led to the loss of subsequent edits without user notification.

Further investigation revealed that the issue occurred only when editing operations caused a splice at specific byte offsets within surrogate pairs. The root cause was identified as the lib0 splice method, which used JavaScript’s .slice() to modify strings containing surrogate pairs. When a splice split a surrogate pair, it resulted in orphaned surrogates that caused URI errors during encoding, halting the sync process silently.

Why It Matters

This bug highlights a critical challenge in handling Unicode characters, especially emojis, in web applications. Since emojis are increasingly common in user content, such issues can lead to data loss and reduce trust in collaborative tools. The incident underscores the importance of robust Unicode handling and error management in real-time editing platforms.

Background

Prior to this discovery, developers suspected network issues or websocket instability as causes of sync failures. The problem was rare and difficult to reproduce, often occurring only with specific emoji operations. The bug emerged as part of ongoing efforts to enhance collaborative editing experiences, which rely heavily on CRDT algorithms and Unicode string manipulation.

Unicode surrogate pairs are a well-known complexity in string processing, but their impact on CRDT-based sync systems was previously underestimated. This incident has prompted a review of string handling practices in such systems.

“This bug exposed the fragility of handling surrogate pairs in our CRDT operations, causing silent data loss during collaborative editing.”

— Lead developer

“Inserting specific emojis triggered the splice that broke the sync, revealing a subtle Unicode handling flaw.”

— Product manager involved in debugging

What Remains Unclear

It remains unclear whether similar issues exist in other parts of the system or with other Unicode characters. The full scope of the bug’s impact across different browsers, devices, and input methods is still being evaluated. Additionally, the long-term solution and whether other CRDT implementations are affected are under investigation.

What’s Next

The development team is working on patches to improve Unicode handling, including safer string manipulation methods that avoid splitting surrogate pairs. They plan to release an update in the coming weeks, along with guidelines for developers on managing Unicode in CRDT-based systems.

Key Questions

What are surrogate pairs in Unicode?

Surrogate pairs are a way to encode characters outside the Basic Multilingual Plane (U+0000 to U+FFFF) in UTF-16. They use two 16-bit code units to represent a single character, such as many emojis.

Why did this bug cause silent data loss?

The splice operation split surrogate pairs, creating orphaned surrogates that the encoding process couldn’t handle, leading to errors that halted synchronization without notifying users.

Is this issue limited to specific emojis?

Yes. It primarily affects emojis that require surrogate pairs, i.e., those above U+FFFF, such as 🤠 or 👩‍🚀. Not all emojis or characters cause this problem.

Will this bug affect other applications?

Potentially. Any application that manipulates Unicode strings with surrogate pairs and relies on similar splice operations could encounter comparable issues.