ngram-core API

The N-gram statistical-text engine — the Model generator. It learns from sample data and produces new, plausible values that were (usually) never in the training set: fast, deterministic, no LLM. You use it directly to train the .ngram models that the model generator then consumes.

Token types

A model is one of three kinds, chosen by what you train it on:

Kind	Constructor	Token	Trains on	Produces	Use
char	train_words	a character	a word list	new words	names, usernames, codes
word	train_phrases	a word	a phrase list	new phrases	company/product names
text	train_text	a word + structure	prose	sentences/paragraphs	bios, reviews

Training

Convenience constructors

let model = NgramModel::train_words(&["Mehmet", "Ayşe", "Mustafa"], 3); // order-3 char model
let model = NgramModel::train_phrases(&["Acme Corp", "Globex Inc"], 2);
let model = NgramModel::train_text(corpus, 3);
// `*_with(items, order, &TokenizerConfig)` variants take an explicit tokenizer.

Streaming / incremental training

For large or remote data, feed batches — the counts form a commutative monoid, so any batch order (or merged workers) yields a byte-identical model:

let mut b = ModelBuilder::words(3, &TokenizerConfig::default());
b.feed_items(&["mehmet", "ayse"]);
b.feed_items(&["mustafa"]);
let model = b.finalize();
// also: ModelBuilder::{phrases, text}, .min_count(n), .merge(other), .feed_text(&str)

.min_count(n) (a builder option) prunes transitions seen fewer than n times — it suppresses verbatim leakage and shrinks the model.

Generation

All generation takes an explicit &mut SplitMix64 (the portable PRNG), so it's deterministic given a seed.

let mut rng = SplitMix64::new(2026);
let c = Constraints::default();

model.word(&mut rng, &c);                       // a single word
model.words(&mut rng, 5, &c);                   // Vec<String>
model.real_word(&mut rng, Some("A"));           // an actual training item (optional prefix)
model.sentence(&mut rng, Some(8), ".", None);   // num_words, punctuation, starts_with
model.text(&mut rng, 200, "…", None);           // max_chars, suffix, starts_with
model.paragraph(&mut rng, Some(4), None);       // num_sentences, starts_with
model.poem(&mut rng, 4, 4, 8);                  // verses, stanza_length, max_words
model.acrostic(&mut rng, "PHONY", 8);           // initials, max_words

Mode	Method	Output
word	word / words	a novel word
sentence	sentence / sentences	a sentence
text	text	truncated prose
paragraph	paragraph	a paragraph
poem	poem	stanzas
acrostic	acrostic	an acrostic
real_word	real_word	an actual training item (frequency-weighted)

Constraints (per-call filters)

struct Constraints {
    starts_with: Option<String>,
    ends_with: Option<String>,
    contains: Option<String>,
    min_length: Option<usize>,
    max_length: Option<usize>,
    exclude_originals: bool, // never emit a value that was in the training set
}

Sampling policy (set on the model)

model.sampling = Sampling { temperature: 1.0, top_k: None, top_p: None, no_repeat: false };

temperature == 1.0 (the default) keeps the exact-integer, cross-runtime reproducible path; other values trade strict reproducibility for diversity.

Anti-memorisation

model.set_verbatim_guard(Some(8)); // reject output sharing a >8-char run with training

Determinism & keyed mapping

model.generate_unique(seed, count, &c);  // distinct values (for unique/PK columns)
model.word_for(key, &c);                 // same key → same value (referential integrity)

word_for uses a portable FNV-1a hash, so a foreign key anonymised this way stays joinable.

The .ngram format

A .ngram file is a portable binary (magic PHNYNG03, gzipped): a documented little-endian, LEB128-varint layout that Rust, PHP, and JS can all read with plain integer reads. This is the cross-runtime contract.

model.save("names.ngram")?;
let model = NgramModel::load("names.ngram")?;
let bytes = model.to_bytes()?;                  // in-memory
let model = NgramModel::from_bytes(&bytes)?;    // rejects anything without the magic

PRNG

let mut rng = SplitMix64::new(seed);
let n = rng.below(100); // uniform in [0, 100)

The PRNG, the format, and inverse-CDF sampling are all part of the portable spec so other runtimes reproduce identical output.