Question about Detection Score

[rianrajagede]

Hi, I'm new to this topic and have a question about the detection score.
When running the demo file with KGW scheme (default parameters) on the C4 dataset without any attack, I'm getting a 100% True Positive Rate (TPR).

Based on my reading of watermarking papers, achieving 100% TPR for detection seems unusually high. Could you please help me understand if this is expected behavior, or if there might be an issue with my understanding of either the watermarking scheme or MarkLLM usage?

Thank you for your assistance.

[panly2003]

This is completely normal behavior. Most text watermarking algorithms can achieve >0.99 TPR in clean cases (without any attacks). Only when facing various attacks will the detection rate decrease.

[rochanaph]

Hi @panly2003 , maybe a related question.

So in detection phase, do existed watermarking algorithms employ the z-score threshold for the same set of data or different set (test)?

Sorry, I'm still puzzled how in real world implementation the dynamic threshold implemented?

[panly2003]

Dynamic thresholds are only used in research for fair comparison, where we can fix the FPR to compare TPR and F1 scores, etc. In real-world implementations, a fixed threshold must be used. 😊