Formal language

About: Formal language is a research topic. Over the lifetime, 5763 publications have been published within this topic receiving 154114 citations.

Theoretical Limitations of Self-Attention in Neural Sequence Models

Abstract: Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.

An Introduction to Formal Languages and Automata, Fifth Edition

Abstract: Written to address the fundamentals of formal languages, automata, and computabilty, An Introduction to Formal Languages and Automata provides an accessible, student-friendly presentation of all material essential to an introductory Theory of Computation course It is designed to familiarize students with the foundations and principles of computer science and to strengthen the students' ability to carry out formal and rigorous mathematical arguments In the new Fifth Edition, Peter Linz continues to offer a straightforward, uncomplicated treatment of formal languages and automata and avoids excessive mathematical detail so that students may focus on and understand the underlying principles In an effort to further the accessibility and comprehension of the text, the author has added new illustrative examples and exercises throughout New and Key Features of the revised and updated Fifth Edition: Includes a new chapter within the appendices on finite-state transducers, including basic results on Mealy and Moore machines This optional chapter can be used to prepare students for further related study Provides an introduction to JFLAP, also within the appendices Many of the exercises in the text require creating structures that are complicated and that have to be tested for correctness JFLAP can greatly reduce students time spent on testing as well as help them visualize abstract concepts A CD-ROM accompanies every new copy of the text and contains the following: A summary description of JFLAP; Numerous new exercises that illustrate the value and efficiency of JFLAP; JFLAP implementations of most of the examples in the text that allow students and instructors to experiment dynamically with these examples

Storage and computation in the mental lexicon

Abstract: This chapter shows that the pocket calculator provides a fundamentally flawed metaphor for understanding morphological structure and processing in the mental lexicon. To this end, it surveys the evidence from experimental studies of lexical processing, and then considers the fine phonetic detail that is present in the acoustic signal. The chapter provides an indication of the kind of formal mathematical model that may help to understand the process and representation better in the mental lexicon. It also discusses theory of speech production, WEAVER models, phonetic evidence, and Hierarchical Temporal Memory (HTM). Whereas the mathematics of formal languages has been a key source of inspiration for morphological theory and models of the mental lexicon, new advances at the intersection of statistics, information science and the neurosciences such as HTM can constitute an important source of inspiration for research on the mental lexicon during the coming years.Keywords: acoustic signal; Lieber; mathematical model; mental lexicon; pocket calculator; Selkirk

On Active Deductive Databases: The Statelog Approach

Abstract: After briefly reviewing the basic notions and terminology of active rules and relating them to production rules and deductive rules, respectively, we survey a number of formal approaches to active rules. Subsequently, we present our own state-oriented logical approach to active rules which combines the declarative semantics of deductive rules with the possibility to define updates in the style of production rules and active rules. The resulting language Statelog is surprisingly simple, yet captures many features of active rules including composite event detection and different coupling modes. Thus, it can be used for the formal analysis of rule properties like termination and expressive power. Finally, we show how nested transactions can be modeled in Statelog, both from the operational and the model-theoretic perspective.