The Unicode Cookbook for Linguists
Free

The Unicode Cookbook for Linguists

By Steven MoranMichael Cysouw
Free
Book Description

This text is a practical guide for linguists, and programmers, who work with data in multilingual computational environments. We introduce the basic concepts needed to understand how writing systems and character encodings function, and how they work together at the intersection between the Unicode Standard and the International Phonetic Alphabet. Although these standards are often met with frustration by users, they nevertheless provide language researchers and programmers with a consistent computational architecture needed to process, publish and analyze lexical data from the world's languages. Thus we bring to light common, but not always transparent, pitfalls which researchers face when working with Unicode and IPA. Having identified and overcome these pitfalls involved in making writing systems and character encodings syntactically and semantically interoperable (to the extent that they can be), we created a suite of open-source Python and R tools to work with languages using orthography profiles that describe author- or document-specific orthographic conventions. In this cookbook we describe a formal specification of orthography profiles and provide recipes using open source tools to show how users can segment text, analyze it, identify errors, and to transform it into different written forms for comparative linguistics research. This book is a prime example of open publishing as envisioned by Language Science Press. It is open access, has accompanying open source software, has open peer review, versioning and so on. Read more in this blog post.

Table of Contents
  • Preface
  • Acknowledgments
  • 1 Writing systems
    • 1.1 Introduction
    • 1.2 Encoding
    • 1.3 Linguistic terminology
  • 2 The Unicode approach
    • 2.1 Background
    • 2.2 The Unicode Standard
    • 2.3 Character encoding system
    • 2.4 Grapheme clusters
  • 3 Unicode pitfalls
    • 3.1 Wrong it ain't
    • 3.2 Pitfall: Characters are not glyphs
    • 3.3 Pitfall: Characters are not graphemes
    • 3.4 Pitfall: Missing glyphs
    • 3.5 Pitfall: Faulty rendering
    • 3.6 Pitfall: Blocks
    • 3.7 Pitfall: Names
    • 3.8 Pitfall: Homoglyphs
    • 3.9 Pitfall: Canonical equivalence
    • 3.10 Pitfall: Absence of canonical equivalence
    • 3.11 Pitfall: Encodings
    • 3.12 Pitfall: File formats
    • 3.13 Pitfall: Incomplete implementations
    • 3.14 Recommendations
  • 4 The International Phonetic Alphabet
    • 4.1 Brief history
    • 4.2 Premises and principles
    • 4.3 IPA encodings
    • 4.4 The need for a single multilingual environment
  • 5 IPA meets Unicode
    • 5.1 The twain shall meet
    • 5.2 Pitfall: No complete IPA code block
    • 5.3 Pitfall: IPA homoglyphs in Unicode
    • 5.4 Pitfall: Homoglyphs in IPA
    • 5.5 Pitfall: Multiple encoding options in IPA
    • 5.6 Pitfall: Tie bar
    • 5.7 Pitfall: Ligatures and digraphs
    • 5.8 Pitfall: Missing decomposition
    • 5.9 Pitfall: Different notions of diacritics
    • 5.10 Pitfall: No unique diacritic ordering
    • 5.11 Pitfall: Revisions to the IPA
    • 5.12 Additions to the IPA
    • 5.13 Unicode IPA Recommendations
  • 6 Practical recommendations
    • 6.1 Unicode
    • 6.2 IPA
    • 6.3 For programmers and potential programmers
  • 7 Orthography profiles
    • 7.1 Characterizing writing systems
    • 7.2 Informal description
    • 7.3 Formal specification
  • 8 Implementation
    • 8.1 Overview
    • 8.2 How to install Python and R
    • 8.3 Python package: segments
    • 8.4 R library: qlcData
    • 8.5 Recipes online
  • References
  • Index
    • Name index
    No review for this book yet, be the first to review.
      No comment for this book yet, be the first to comment
      You May Also Like