The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.

https://dl.acm.org/doi/pdf/10.1145/3442188.3445922

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? &#x1f99c;

Each copy of a text document can be made different in a nearly invisible way by repositioning or modifying the appearance of different elements of text, i.e., lines, words, or characters. A unique copy can be registered with its recipient, so that subsequent unauthorized copies that are retrieved can be traced back to the original owner. In this paper we describe and compare several mechanisms for marking documents and several other mechanisms for decoding the marks after documents have been subjected to common types of distortion. The marks are intended to protect documents of limited value that are owned by individuals who would rather possess a legal than an illegal copy if they can be distinguished. We will describe attacks that remove the marks and countermeasures to those attacks. An architecture is described for distributing a large number of copies without burdening the publisher with creating and transmitting the unique documents. The architecture also allows the publisher to determine the identity of a recipient who has illegally redistributed the document, without compromising the privacy of individuals who are not operating illegally. Two experimental systems are described. One was used to distribute an issue of the IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, and the second was used to mark copies of company private memoranda.

Copyright protection for the electronic distribution of text documents : Identification and protection of multimedia information

During the recent years, the issue of preserving the integrity of digital text has become a focus of interest in the transmission of online content on the Internet. Watermarking has a useful tool in the protection of digital text content as it solves the problem of tampering, duplicating, unauthorized access, and security breaches. The rapid development currently observable in information transfer and access is the consequences of the widespread usage of the Internet. When it comes to the different types of digital data, text constitutes the most complex and challenging type to which the method of text watermarking can be applied. Text watermarking constitutes a highly complex task, most of all, since only limited research has been done in this field. In order to ensure the successful evaluation, analysis, and implementation, a comprehensive research needs to be performed. This paper studies the theory, methods, and applications of text watermarking, which includes the discussion on the definition, embedding and extracting processes, requirements, approaches, and language applications of the established text watermarking methods. This paper reviews in detail the new classification of text watermarking, which is through embedding process and its related issues of attacks and language applicability. Open research challenges and future directions are also investigated, with a focus on its information integrity, information availability, originality preservation, information confidentiality, protection of sensitive information, document transformation, cryptography application, and language flexibility.

A Review of Text Watermarking: Theory, Methods, and Applications

A comprehensive survey on robust image watermarking

With the popularity of smartphones and widespread use of high-speed Internet, social media has become a vital part of people’s daily life. Currently, text messages are used in many applications, such as mobile chatting, mobile banking, and mobile commerce. However, when we send a text message via short message service (SMS) or social media, the information contained in the text message transmits as a plain text, which exposes it to attacks. In some cases, this information may be confidential, such as banking credentials, secret missions, and confidential appointments; moreover, it is a major drawback to send such information via SMS or social media, as neither provides security before transmission. In this paper, we propose a novel text steganography technique called AITSteg, which provides end-to-end security during the transmission of text messages via SMS or social media between end users. The AITSteg technique is evaluated by considering a trusted scenario. We then evaluate the efficiency of the proposed technique in terms of embedding capacity, invisibility, robustness, and security. The experiments confirm that the AITSteg is able to prevent various attacks, including man-in-the-middle attacks, message disclosure, and manipulation by readers. Moreover, we compare the experimental results with the existing techniques to show the superiority of the proposed technique. To the best of our knowledge, this is the first text steganography technique that provides end-to-end secure transmission of the text message using symmetric keys via social media.

AITSteg: An Innovative Text Steganography Technique for Hidden Transmission of Text Message via Social Media

Among multimedia content, one of the most predominant medium is text content. There have been lots of efforts to protect and secure text information over the Internet. The limitations of existing works have been identified in terms of watermark capacity, time complexity and memory complexity. In this work, an invisible digital watermarking approach has been proposed to protect and secure the most sensitive text i.e. Digital Holy Quran. The proposed approach works by XOR-ing only those Quranic letters that has certain diacritics associated with it. Due to sensitive nature of Holy Quran, diacritics play vital role in the meaning of the particular verse. Hence, securing letters with certain diacritics will preserve the original meaning of Quranic verses in case of alternation attempt. Initial results have shown that the proposed approach is promising with less memory complexity and time complexity compared to existing approaches.

/pdf/associated-diacritical-watermarking-approach-to-protect-2a7ifpaq2k.pdf

Nurul Shamimi Kamaruddin

Papers

A Review of Text Watermarking: Theory, Methods, and Applications

Associated diacritical watermarking approach to protect sensitive arabic digital texts