Data Generation Architecture

Scope: This section covers the core data generation system - PGDL schema language, generator types, N-gram models, and the git-distributed package format (phony.json). For cloud platform features (sync, mock API, snapshots), see Cloud Platform Architecture.

Implementation status (built engine)

This section documents the original seven-type design. In the built engine the taxonomy is five core generators (Logic, List, Model, Statistical, Event Sequence) plus composition (the universal composer). Template and Linked are folded into composition: template → a composition with a body (and optional weighted variants); linked → either a list over an asset of coherent objects, or a composition output over prior let steps.

Overview

Phony's data generation architecture is built on the principle of "Data Generation as Code" - treating synthetic data generation with the same rigor as infrastructure management.

┌─────────────────────────────────────────────────────────────────────────┐
│                    DATA GENERATION ARCHITECTURE                          │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│                         PGDL Schema                                       │
│                     (schema.pgdl.json)                                    │
│                            │                                             │
│    ┌───────────────────────┼───────────────────────┐                    │
│    ▼           ▼           ▼           ▼           ▼                    │
│ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐                      │
│ │ Logic │ │ List  │ │ Model │ │Statis-│ │Linked │                      │
│ │       │ │       │ │(N-gram│ │tical  │ │       │                      │
│ │UUIDs, │ │Codes, │ │Names, │ │Distri-│ │City+  │                      │
│ │Numbers│ │Enums  │ │Text   │ │butions│ │Country│                      │
│ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘                      │
│     └─────────┴─────────┼─────────┴─────────┘                          │
│                         ▼                                               │
│    ┌────────────────────────────────────────┐                          │
│    │          Template Generator            │                          │
│    │    (Compose, Format, Operations)       │                          │
│    └────────────────────┬───────────────────┘                          │
│                         │                                               │
│    ┌────────────────────┼────────────────────┐                         │
│    ▼                    ▼                    ▼                          │
│ ┌─────────┐      ┌─────────────┐      ┌─────────┐                      │
│ │ Event   │      │ Cross-Table │      │ Privacy │                      │
│ │Sequence │      │ Operations  │      │ Features│                      │
│ │         │      │             │      │         │                      │
│ │Chrono-  │      │Sum, Count,  │      │Diff Priv│                      │
│ │logical  │      │Aggregates   │      │Geo-Anon │                      │
│ └────┬────┘      └──────┬──────┘      └────┬────┘                      │
│      └──────────────────┼──────────────────┘                           │
│                         ▼                                               │
│                 ┌──────────────┐                                        │
│                 │    Phony     │                                        │
│                 │   Package    │                                        │
│                 │  (git repo)  │                                        │
│                 └──────────────┘                                        │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Core Components

Component	Description	Documentation
Generator Types	Five core generators (Logic, List, Model, Statistical, Event Sequence) + composition — documented via the original seven-type design	Generator Types
Advanced Concepts	Consistency, linking, statistical generation, differential privacy, format-preserving transformation	Advanced Concepts
N-gram Models	Statistical text generation using Markov chains	N-gram Models
PGDL Specification	Declarative JSON schema for data generation	PGDL Specification
Expression Language	Template syntax (PEL) for composition	Expression Language
Package Manager	Git-based distribution — phony.json, MVS resolution, content-addressed store, phony.lock	Package Manager
Package Format	Historical .phony tarball design — superseded by the git-based Package Manager	Package Format
Locale System	Multi-layer inheritance for i18n	Locale System
Execution Model	OSS vs Cloud runtime differences	Execution Model
Generator × Asset Matrix	The full combination contract: every generator, asset, locale, and repo-split pattern	Generator × Asset Matrix

Key Principles

1. Declarative - Define WHAT data you need, not HOW to generate it
2. Portable - The same git-distributed package (phony.json + assets) runs on every runtime
3. Composable - Mix N-gram models + templates + lists + logic seamlessly
4. Deterministic - Same seed produces same output across all runtimes
5. Consistent - Same input produces same output across tables and databases
6. Statistically Accurate - Generated data matches real-world distributions
7. Privacy-Preserving - Differential privacy (continuous, built), min_count pruning + verbatim guard (built); k-anonymity/geo masking are cloud-platform target design
8. Relationship-Aware - Coherent multi-field records via list-over-object-assets / composition output (the former "linked" concept)
9. Open - OSS core with MIT license, Cloud for enterprise features