/pdf/understanding-u-s-regional-linguistic-variation-with-twitter-4egiscnxk3.pdf

Understanding U.S. regional linguistic variation with Twitter data analysis

Genetically related languages originate from a common proto-language. In the absence of historical records, proto-languages have to be reconstructed from surviving cognates, that is words that existed in the proto-language and are still present in some form in its descendants. The language reconstruction methods have so far been largely based on informal and intuitive criteria. In this thesis, I present techniques and algorithms for performing various stages of the reconstruction process automatically. 
The thesis is divided into three main parts that correspond to the principal steps of language reconstruction. The first part presents a new algorithm for the alignment of cognates, which is sufficiently general to align any two phonetic strings that exhibit some affinity. The second part introduces a method of identifying cognates directly from the vocabularies of related languages on the basis of phonetic and semantic similarity. The third part describes an approach to the determination of recurrent sound correspondences in bilingual wordlists by inducing models similar to those developed for statistical machine translation. 
The proposed solutions are firmly grounded in computer science and incorporate recent advances in computational linguistics, articulatory phonetics, and bioinformatics. The applications of the new techniques are not limited to diachronic phonology, but extend to other areas of computational linguistics, such as machine translation.

/pdf/algorithms-for-language-reconstruction-37mbqyduxc.pdf

Algorithms for language reconstruction

The organizing concept behind dialect variation is still seen predominantly as the areas within which similar varieties are spoken. The opposing view—that dialects are organized in a continuum without sharp boundaries—is likewise popular. This article introduces a new element into the discussion, which is the opportunity to view dialectal differences in the aggregate. We employ a dialectometric technique that provides an additive measure of pronunciation difference: the (aggregate) pronunciation distance. This allows us to determine how much of the linguistic variation is accounted for by geography. In our sample of 27 Dutch towns and villages, the variation ranges between 65% and 81%, which lends credence to the continuum view. The borders of well-established dialect areas nonetheless show large deviations from the expected aggregate pronunciation distance. We pay particular attention to a puzzle concerning the subjective perception of continua introduced by Chambers and Trudgill (1998): a traveller walking in a straight line from village to village notices successive small changes, but seldom, if ever, observes large differences. This sounds like a justification of the continuum view, but there is an added twist. Might the traveller be misled by the perspective of most recent memory? We use the Chambers–Trudgill puzzle to organize our argument at several points.

Dialect Areas and Dialect Continua

We describe ongoing work in the experimental evaluation of a range of methods for measuring the phonetic distance between the dialectal variants of pronunciations. Alllare variants of Levenshtein distance, both simple (based on atomic characters) and complex (based on feature vectors). The measurements using feature vectors varied according to whether city-block distance, Euclidean distance or (a measure using) Pearson's correlation coefficient was taken as basic. Variants of these using feature weighting by entropy reduction were systematically compared, as was the representation of diphthongs (as one symbol or two). The results were compared to well-established scholarship in dialectology, yielding a Calibration of the method. These results indicate that feature representations are more sensitive, that city-block distance is a good measure of phonetic overlap of feature vectors, that weighting is not useful, and that two-phone representations of diphthongs provide a more satisfactory base for this sort of comparison. K e y w o r d s : dialectology, phonetic (dis)similarity 1 M o t i v a t i o n Dialectologists frequently speak of the range of dialects they describe as a "continuum", 1 which suggests a need t0 supersede the inherently discrete method of isogl0sses. Dialectologists have long recognized the need for alternative notions of dialectal relationships (Durand (1889),p.49). 1For example,.Tait ol/Inuit: ';a fairly unbroken chain of dialects [...] the furthest extremes of the continuum being unintelligib!e to one another" (Tait (1994), p.3) It is furthermore the case that a sensitive measure of dialectal distance could have broad application to questions in sociolinguistics and historical linguistics, e.g. the significance of political boundaries, the effect of the media, etc. Levenshtein distance is a measure of string distance that has been applied to problems in speech recognition, bird song ethology, and genetics. It is presented in (Kruskal, 1983), and may be understood as the cost of (the least costly set of) operations mapping from one string to another. Kessler (1995) applied Levenshtein distance to Irish Gaelic dialects with remarkable success, and Nerbonne et al. (1996) extended the application of his techniques to Dutch dialects, similarly with respectable results. Although Kessler and Nerbonne et al. (1996) experimented with more sensitive measures, their best results were based on calculations of phonetic distance in which phonetic overlap was binary: nonidentical phones contribute to phonetic distance, identical ones do not. Thus the pair [a,t] count as different to the same degree as 2 B a c k g r o u n d In the interest of space we omit an introduction to Levenshtein distance, referring to (Kruskal, 1983). It may be understood as the cost of (the least costly set of) operations mapping from one string to another. The basic costs are those of (single-phone) insertions and deletions, each of which costs half that of substitutions. Nerbonne et al. (1996)explains its use in the present application at some length. The various modifications below all tinker with the cost of substituting one phone for another. Kessler (1995) experimented with making the measure more sensitive, but found little progress in using features, for example. The present paper experiments systematically with several variations on the basic Levenshtein theme.

Measuring Dialect Distance Phonetically

Dialectvariatie beter in beeld te brengen Het centrale thema van Martijn Wielings proefschrift is het onderzoeken van dialectvariatie. Om een objectief beeld te krijgen van dialectvariatie meten we dialectafstanden op basis van honderden woorden automatisch op basis van genoteerde uitspraken. Wieling verfijnde deze afstandsmaat door het gebruik van (automatisch bepaalde) akoestisch gevoelige klankafstanden. Daarnaast introduceert hij een nieuwe methode die het mogelijk maakt om groepen van vergelijkbare dialecten te vinden, waarbij tegelijkertijd de onderliggende taalkundige basis (gebaseerd op klankcorrespondenties) wordt bepaald. Voor zowel Engelse als Nederlandse dialecten vond Wieling aannemelijke geografische dialectgebieden samen met hun meest karakteristieke klankcorrespondenties. In het Fries is bijvoorbeeld de toevoeging van ‘sj’ erg kenmerkend: ‘wachten’ wordt ‘wachtsje’. Wieling ontwikkelde een integrale aanpak, die de invloed van diverse factoren op dialectvariatie per woord kan bepalen. Niet alleen kan via deze methode gekeken worden naar de invloed van geografische ligging, maar ook naar de rol van verschillende sociale en woord-gerelateerde factoren (zoals leeftijd van de spreker en woordfrequentie). Wieling ontdekte bijvoorbeeld bij Nederlandse dialectwoorden dat deze het meest verschillen van de standaardtaal in de provincies Friesland, Groningen, Drenthe, Overijssel, Limburg en Zeeland. Daarnaast blijkt dat de dialectuitspraken van een gemeenschap met een klein aantal inwoners of een hoge gemiddelde leeftijd meer afwijken van de Nederlandse standaardtaal dan die van een gemeenschap met een groot aantal inwoners of een lage gemiddelde leeftijd. Ook blijkt dat meer frequente woorden meer resistent zijn tegen standaardisatie (ook in andere talen). Wielings dissertatie gaat specifiek over uitspraken in dialect. Het zegt niets over het accent van deze sprekers in de standaardtaal.

A quantitative approach to social and geographical dialect variation

Traditional dialectology relies on identifying language features which are common to one dialect area while distinguishing it from others. It has difficulty in dealing with partial matches of features and with nonoverlapping language patterns. This paper applies Levenshtein distance—a measure of string distance—to pronunciations to overcome both of these difficulties. Partial matches may be quantified, and nonoverlapping patterns may be included in weighted averages of phonetic distance. The result accords with traditonal dialectology to a satisfying degree.

E van den Hout

Papers

Phonetic Distance between Dutch Dialects