languagechange package

Subpackages

• languagechange.models package
  • languagechange.models.change package
    • languagechange.models.change.metrics module
    • languagechange.models.change.timeseries module
    • languagechange.models.change.widid module
    • Module contents
  • languagechange.models.meaning package
    • languagechange.models.meaning.clustering module
    • languagechange.models.meaning.meaning module
      • MeaningModel
      • WordSenseInduction
      • StaticEmbedding
      • SGNS
    • Module contents
  • languagechange.models.representation package
    • languagechange.models.representation.alignment module
      • OrthogonalProcrustes
    • languagechange.models.representation.contextualized module
      • generate_cache_key()
      • ContextualizedModel
      • ContextualizedEmbeddings
      • XL_LEXEME
      • BERT
      • RoBERTa
    • languagechange.models.representation.definition module
      • Message
      • DefinitionGenerator
      • LlamaDefinitionGenerator
      • DefinitionOutput
      • ChatModelDefinitionGenerator
      • T5DefinitionGenerator
    • languagechange.models.representation.prompting module
    • languagechange.models.representation.static module
      • RepresentationModel
      • StaticModel
      • CountModel
      • PPMI
      • SVD
      • RandomIndexing
    • Module contents
  • Module contents

Submodules

languagechange.benchmark module

languagechange.cache module

Cache manager with atomic write helpers for file-based caches.

class languagechange.cache.CacheManager(cache_dir=None)

Bases: object

Manages cache files with atomic write operations to prevent data corruption in concurrent environments. The cache files are saved to a directory that can be specified during initialization.

atomic_write(path)

Provides a context manager for writing to cache files in an atomic way. This ensures that partial writes do not corrupt the target file, especially when multiple processes or threads access the same file.

Parameters:

path (str) – The relative path to the cache file within the cache directory.

Yields:

file object –

A writable file object for writing data. The file is temporary

and will be renamed to the target file path after the write operation completes successfully.

languagechange.corpora module

Corpus utilities for line-level corpora and search helpers.

class languagechange.corpora.Line(raw_text=None, tokens=None, lemmas=None, pos_tags=None, fname=None, raw_lemma_text=None, raw_pos_text=None, **kwargs)

Bases: object

Wraps a corpus line with token, lemma, and POS metadata.

tokens()

lemmas()

pos_tags()

tokens_by_feature(feat=<class 'str'>)

raw_text()

raw_lemma_text()

raw_pos_text()

raw_text_by_feature(feat='token')

search(search_term, time=None)

Searches the line given a search_term.

Parameters:

search_term (SearchTerm) – SearchTerm

Return type:

TargetUsageList

Returns: A TargetUsageList of all matches.

class languagechange.corpora.Corpus(name, language=None, time=no time specification, time_function=None, skip_lines=0, **args)

Bases: object

Base interface for corpora that support search and tokenization.

set_sentences_iterator(sentences)

search(search_terms)

Searches through the corpora by calling Line.search() on all lines.

Parameters:

search_terms (List[str | Pattern | SearchTerm]) – List[ str | Pattern | SearchTerm ] If a search term is str or Pattern it is converted to a SearchTerm and matches tokens only SearchTerm(word_feature = ‘token’).

Return type:

UsageDictionary

Returns: A UsageDictionary containing all search results for each search term.

tokenize(tokenizer='trankit', split_sentences=False, batch_size=128)

Yield tokenized sentences using Trankit, optionally splitting sentences.

Parameters:

• tokenizer (str, optional) – Tokenizer backend. Defaults to “trankit”.

• split_sentences (bool, optional) – Split paragraphs into sentences. Defaults to False.

• batch_size (int, optional) – Number of lines to accumulate before processing. Defaults to 128.

lemmatize(lemmatizer='trankit', pretokenized=False, tokenize=False, split_sentences=False, batch_size=128)

pos_tagging(pos_tagger='trankit', pretokenized=False, tokenize=False, split_sentences=False, batch_size=128)

tokens_lemmas_pos_tags(nlp_model='trankit', tokens=True, split_sentences=False, batch_size=128)

segment_sentences(segmentizer='trankit', batch_size=128)

folder_iterator(path)

cast_to_vertical(vertical_corpus)

save()

save_tokenized_corpora(tokens=True, lemmas=False, pos=False, save_format='linebyline', file_specification=None, file_ending='.txt', tokenizer='trankit', lemmatizer='trankit', pos_tagger='trankit', split_sentences=True, batch_size=128)

Parameters:

corpora (Self | List[Self])

class languagechange.corpora.LinebyLineCorpus(path, **kwargs)

Bases: Corpus

line_iterator()

class languagechange.corpora.VerticalCorpus(path, sentence_separator='\n', field_separator='\t', field_map={'lemma': 1, 'pos_tag': 2, 'token': 0}, **args)

Bases: Corpus

line_iterator()

class languagechange.corpora.XMLCorpus(path, sentence_tag='sentence', token_tag='token', is_lemmatized=False, lemma_tag=None, is_pos_tagged=False, pos_tag_tag=None, text_tag='text', **args)

Bases: Corpus

get_attribute(tag, attribute)

line_iterator()

cast_to_linebyline(linebyline_corpus)

Parameters:

linebyline_corpus (LinebyLineCorpus)

cast_to_vertical(vertical_corpus)

Parameters:

vertical_corpus (VerticalCorpus)

class languagechange.corpora.SprakBankenCorpus(path, sentence_tag='sentence', token_tag='token', is_lemmatized=True, lemma_tag='lemma', is_pos_tagged=True, pos_tag_tag='pos', **args)

Bases: XMLCorpus

get_attribute(tag, attribute)

class languagechange.corpora.HistoricalCorpus(*args, **kwargs)

Bases: SortedKeyList

line_iterator()

Iterates through all of the corpora, and yields all of the lines that are possible to get.

search(search_terms, index_by_corpus=False)

Searches through all of the corpora by calling search() for each of them.

Parameters:

• search_terms (List[str | Pattern | SearchTerm]) – List[ str | Pattern | SearchTerm ] If search term is str or Pattern it is converted to a SearchTerm and matches tokens only SearchTerm(word_feature = ‘token’).

• index_by_corpus – bool, default False decides whether the usages for a given word should be a dictionary, with keys as the corpus names and values as lists of usages, or a list of all usages across corpora.

Returns: a dictionary containing all search results from the included corpora.

languagechange.evaluation module

languagechange.resource_manager module

Resource manager that downloads and caches datasets and models.

class languagechange.resource_manager.LanguageChange

Bases: object

load_resources_hub()

Refresh the resource hub index from GitHub.

download_ui()

Interactive prompt for selecting and downloading resources.

download(resource_type, resource_name, dataset, version)

Download and cache a resource from the resource hub.

get_resource(resource_type, resource_name, dataset, version)

Return the path to a cached resource, downloading it if necessary.

save_resource(resource_type, resource_name, dataset, version)

Create a local resource directory for persistent storage.

languagechange.search module

Helper utilities for searching corpora for target terms.

languagechange.search.expand_dictionary(words)

Placeholder for future dictionary expansion utilities.

Parameters:

words (List[str]) – Words to expand into additional search terms.

class languagechange.search.SearchTerm(term, regex=False, word_feature='token')

Bases: object

Describes a search target and the features to scan within a corpus line.

Parameters:

• term (str)

• regex (bool)

• word_feature (str | Set)

VALID_WORD_FEATURES = ['lemma', 'token', 'pos']

languagechange.usages module

Target usage helpers and containers for LanguageChange.

class languagechange.usages.POS(*values)

Bases: Enum

Enumeration of supported parts of speech for targets.

NOUN = 1

VERB = 2

ADJECTIVE = 3

ADVERB = 4

class languagechange.usages.Target(target)

Bases: object

Stores a target word together with optional metadata.

Parameters:

target (str)

set_lemma(lemma)

Parameters:

lemma (str)

set_pos(pos)

Parameters:

pos (POS)

class languagechange.usages.TargetUsage(text, offsets, time=None, **kwargs)

Bases: object

Represents an individual usage with offsets and optional time metadata.

Parameters:

• text (str)

• offsets (str)

• time (Time)

text()

start()

end()

time()

to_dict()

class languagechange.usages.DWUGUsage(target, date, grouping, identifier, description, **args)

Bases: TargetUsage

DWUG-specific usage metadata, including annotator judgments.

class languagechange.usages.TargetUsageList(iterable=(), /)

Bases: list

List of TargetUsage instances with serialization helpers.

save(path, target)

load(target)

time_axis()

to_dict()

class languagechange.usages.UsageDictionary

Bases: dict

Dictionary mapping words to TargetUsageList instances.

save(path, words={})

load(path, words={})

languagechange.utils module

Simple time representations used across the LanguageChange toolkit.

class languagechange.utils.Time

Bases: object

class languagechange.utils.LiteralTime(time)

Bases: Time

Represents a literal timestamp or label for usage references.

Parameters:

time (str)

class languagechange.utils.NumericalTime(time)

Bases: Time

Numeric timestamp (e.g., time slice) that supports comparisons.

Parameters:

time (Number)

class languagechange.utils.TimeInterval(start, end)

Bases: Time

Represents an interval between two Time points.

Parameters:

• start (Time)

• end (Time)

Module contents

Core exports for the LanguageChange toolkit.