Implementing Watermarking for Language Models [P]

[u/Saad_ahmed04]

I recently implemented a minimal, educational version of SynthID-Text-style watermarking for language models.

I saw anthropic post about how they'll start adding watermarks to their model responses and it made me very curious as to how they'll do it and what do they even mean by watermark here. Like will we start getting random ads or something in the middle of model responses or what.

Then decided to read their article and found out that watermark is not a visible message at all. It is a subtle statistical pattern introduced while the model chooses its tokens.

My implementation is not an exact reproduction of the original SynthID-Text system. I simplified or implemented a few components differently to keep the project understandable, but the main idea is there I think.

Github: https://github.com/Saad1926Q/llm-watermark

If you find it interesting then you may star the repo !!

[u/Lopsided-Bridge-9810]

Reliable watermarking for linguistically based inferencing is not possible, imo. You can feed the output to multiple other llms that don't watermark and ask to have text rewritten a certain way, you can translate the text to other languages and translate back and the entire statistic signature gets lost. Linguistic styled can be targeted and attacked. There is no watermarking happening in text based llms anytime soon. I mean labs can watermark but it's a nonsense approach.

[u/schubidubiduba → u/Lopsided-Bridge-9810]

A recent benchmark showed that even after backtranslation using Chinese as intermediate language, watermarking accuracy remains at 70%. Whether that is high enough to be useful is a different question.

[u/Lopsided-Bridge-9810 → u/schubidubiduba]

I think that only remains this high when watermarking is not mentioned and the llm is not specially instructed?

[u/schubidubiduba → u/Lopsided-Bridge-9810]

Yes this is for standard translation. I don't remember which model or prompt, or if it even was an LLM.