tessl/golang-golang-org-x--text

Go supplementary packages for text processing, many involving Unicode

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Securityby

Pending

The risk profile of this skill

Overview

Eval results

Files

golang.org/x/text

Name: tessl/golang-golang-org-x--text
Author: tessl

The golang.org/x/text package provides comprehensive internationalization (i18n) and localization (l10n) functionality for Go applications. It includes text transformation, character encoding conversion, Unicode normalization, language-sensitive operations, and locale-specific formatting.

Package Information

Package Name: golang.org/x/text
Package Type: golang
Language: Go
Installation: go get golang.org/x/text

Core Imports

import (
    "golang.org/x/text/language"
    "golang.org/x/text/message"
    "golang.org/x/text/transform"
    "golang.org/x/text/encoding/charmap"
    "golang.org/x/text/cases"
    "golang.org/x/text/collate"
    "golang.org/x/text/number"
)

Basic Usage

package main

import (
    "fmt"
    "golang.org/x/text/language"
    "golang.org/x/text/message"
    "golang.org/x/text/number"
    "golang.org/x/text/cases"
)

func main() {
    // Language tag parsing and matching
    tag := language.Make("en-US")
    fmt.Println(tag) // en-US

    // Localized message printing
    p := message.NewPrinter(language.German)
    p.Printf("There are %d items\n", 1234)

    // Number formatting
    p = message.NewPrinter(language.English)
    fmt.Fprintln(p, number.Decimal(1234567.89))

    // Case mapping
    caser := cases.Title(language.English)
    fmt.Println(caser.String("hello world")) // Hello World
}

Capabilities

This library is organized into 8 major functional areas, each documented in detail in the sub-documents linked below.

Character Encoding

Provides character encoding conversion between UTF-8 and various legacy encodings including IBM code pages, ISO-8859 variants, Windows code pages, and Asian encodings (Shift JIS, EUC-JP, EUC-KR, GBK, Big5).

// Core encoding types
type Encoding interface {
    NewDecoder() *Decoder
    NewEncoder() *Encoder
}

type Decoder interface {
    Bytes(b []byte) ([]byte, error)
    String(s string) (string, error)
    Reader(r io.Reader) io.Reader
}

type Encoder interface {
    Bytes(b []byte) ([]byte, error)
    String(s string) (string, error)
    Writer(w io.Writer) io.Writer
}

Character Encoding Details

Language Tags and Display

Implements BCP 47 language tag parsing, validation, matching, and display names for languages, scripts, and regions in multiple languages.

// Language tag type
type Tag struct {}

// Core functions
func Parse(s string) (Tag, error)
func Make(s string) Tag
func Compose(part ...interface{}) (Tag, error)

// Language matcher for content negotiation
type Matcher interface {
    Match(t ...Tag) (tag Tag, index int, c Confidence)
}

func NewMatcher(t []Tag, options ...MatchOption) Matcher

Language Tags and Display Details

Unicode Operations

Provides bidirectional text support, Unicode normalization (NFC, NFD, NFKC, NFKD), CLDR data access, range table utilities, and rune name lookups.

// Normalization forms
type Form int

const (
    NFC  Form = iota
    NFD
    NFKC
    NFKD
)

func (f Form) String(s string) string
func (f Form) Bytes(b []byte) []byte
func (f Form) IsNormal(b []byte) bool

// Bidirectional text support
type Direction int

const (
    LeftToRight Direction = iota
    RightToLeft
    Mixed
    Neutral
)

Unicode Operations Details

Text Transformation

Provides a general framework for text transformation including rune mapping, filtering, width conversion (half-width/full-width), and case mapping.

// Transform interface
type Transformer interface {
    Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error)
    Reset()
}

// Chain multiple transformers
func Chain(t ...Transformer) Transformer

// Case mapping
type Caser interface {
    Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error)
    String(s string) string
    Bytes(b []byte) []byte
}

func Upper(t language.Tag, opts ...Option) Caser
func Lower(t language.Tag, opts ...Option) Caser
func Title(t language.Tag, opts ...Option) Caser
func Fold(opts ...Option) Caser

Text Transformation Details

Collation and Sorting

Provides language-sensitive string comparison and sorting based on Unicode Collation Algorithm (UCA).

// Collator for string comparison
type Collator struct {}

func New(t language.Tag, o ...Option) *Collator

func (c *Collator) Compare(a, b []byte) int
func (c *Collator) CompareString(a, b string) int
func (c *Collator) Sort(x Lister)
func (c *Collator) SortStrings(x []string)

// Builder for custom collation tables
type Builder struct {}

func NewBuilder() *Builder

Collation and Sorting Details

Number and Currency Formatting

Provides locale-specific formatting for numbers, percentages, and currencies.

// Number formatting
func Decimal(x interface{}, opts ...Option) Formatter
func Percent(x interface{}, opts ...Option) Formatter
func Scientific(x interface{}, opts ...Option) Formatter
func Engineering(x interface{}, opts ...Option) Formatter

// Currency formatting
type Unit struct {}

func ParseISO(s string) (Unit, error)
func (u Unit) Amount(amount interface{}) Amount

// Currency query
type QueryIter interface {
    Next() bool
    Unit() Unit
    Region() language.Region
}

func Query(options ...QueryOption) QueryIter

Formatting Details

Message Localization

Provides a complete localization pipeline including message extraction, translation management, catalog building, and localized message printing with plural support.

// Printer for localized messages
type Printer struct {}

func NewPrinter(t language.Tag, opts ...Option) *Printer

func (p *Printer) Printf(key Reference, a ...interface{}) (n int, err error)
func (p *Printer) Sprintf(key Reference, a ...interface{}) string

// Catalog builder
type Builder struct {}

func NewBuilder(opts ...Option) *Builder

func (b *Builder) Set(tag language.Tag, key string, msg ...Message) error
func (b *Builder) SetString(tag language.Tag, key string, msg string) error

// Plural support
type Form byte

const (
    Other Form = iota
    Zero
    One
    Two
    Few
    Many
)

Message Localization Details

Search and Security

Provides language-sensitive search/matching and security-related text processing including PRECIS profiles for usernames and the Bidi Rule for internationalized domain names.

// Language-sensitive search
type Matcher struct {}

func New(t language.Tag, opts ...Option) *Matcher

func (m *Matcher) IndexString(s, pat string, opts ...IndexOption) (start, end int)
func (m *Matcher) Equal(a, b []byte) bool

// PRECIS profiles for secure text
type Profile struct {}

var (
    Nickname             *Profile
    UsernameCaseMapped   *Profile
    UsernameCasePreserved *Profile
    OpaqueString         *Profile
)

func (p *Profile) String(s string) (string, error)
func (p *Profile) Compare(a, b string) bool

// Bidi Rule validation
func Valid(b []byte) bool
func ValidString(s string) bool

Search and Security Details

Architecture

The golang.org/x/text package is structured around several key design principles:

Transform Pipeline: The transform package provides a unified interface for byte transformations, used by encoding, normalization, case mapping, and other operations.
Language Tags: The language package implements BCP 47 and provides the foundation for all language-sensitive operations.
CLDR Data: Many packages derive their behavior from Unicode CLDR (Common Locale Data Repository) data, ensuring consistency with international standards.
Lazy Loading: Large data tables are often generated at build time and loaded only when needed.
Composability: Transformers can be chained together, allowing complex text processing pipelines to be built from simple components.