Database Sync Architecture
Phony's Database Sync engine synchronizes production databases to non-production environments with automatic PII anonymization — and it does so local-first: all row data is read, transformed, and written inside the customer's own infrastructure by the phony agent. Phony Cloud is a control plane that orchestrates jobs and receives only metadata.
Implementation status
Everything on this page is target design. The Rust generation engine and CLI exist; the sync agent and the cloud control plane are not built yet.
Architecture Overview
The defining split: the data plane (the phony agent) runs next to the customer's databases; the control plane (Phony Cloud) schedules work and collects metadata. Production data never leaves the customer's network.
┌─────────────────────────────────────────────────────────────────────────┐
│ DATABASE SYNC ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ CUSTOMER INFRASTRUCTURE (data plane) PHONY CLOUD (control │
│ ═════════════════════════════════════ plane — metadata only) │
│ ═══════════════════════ │
│ ┌─────────────────┐ local │
│ │ Production DB │ connection ┌──────────────────┐ │
│ │ (MySQL/PG) │◄──────────────►│ PHONY AGENT │ outbound │
│ │ │ │ (Rust binary) │ only │
│ │ 100GB / PII │ │ │──────────────► │
│ └─────────────────┘ │ ┌──────────────┐ │ gRPC/HTTPS │
│ │ │Schema │ │ │
│ │ │Analyzer │ │ ┌────────────┐ │
│ │ │• FK Detection│ │ │ Dashboard │ │
│ │ │• PII Detect. │ │ │ Scheduler │ │
│ │ │• Type Map │ │ │ Job State │ │
│ │ └──────┬───────┘ │ │ PII │ │
│ │ ▼ │ │ Inventory │ │
│ │ ┌──────────────┐ │ │ Audit Log │ │
│ │ │Transform │ │ │ Compliance │ │
│ │ │Pipeline │ │ │ Reports │ │
│ │ │• Stream Read │ │ └────────────┘ │
│ │ │• Batch Xform │ │ │
│ │ │• Par. Write │ │ Receives: │
│ │ └──────┬───────┘ │ • schemas │
│ ┌─────────────────┐ local │ ▼ │ • column │
│ │ Staging DB │ connection │ ┌──────────────┐ │ classific. │
│ │ (MySQL/PG) │◄──────────────►│ │Rust Engine │ │ • job status │
│ │ │ │ │+ local models│ │ • report │
│ │ 100GB / No PII │ │ │5M records/sec│ │ artifacts │
│ └─────────────────┘ │ └──────────────┘ │ │
│ └──────────────────┘ NEVER rows. │
│ │
└─────────────────────────────────────────────────────────────────────────┘Consequences of this split:
- Production data never reaches Phony's servers. The trust barrier that kills anonymization deals ("send us your production DB") does not exist here — that is the selling point, and the KVKK/GDPR posture falls out of it.
- No inbound access to customer infrastructure. The agent makes outbound-only connections to the control plane; there are no tunnels into the customer network from the cloud.
- Models trained on customer data stay in customer infrastructure. The agent trains and uses N-gram models locally; only model metadata (name, schema hash) is reported.
Connection Management
The agent runs next to your database
Because the data plane lives inside the customer's network, "connecting a database" means telling the local agent how to reach it — usually a private hostname on the same VPC or host. Customer databases are never exposed to the public internet, and never to Phony.
┌─────────────────────────────────────────────────────────────────────────┐
│ CONNECTION MODEL │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ DEFAULT: Direct local access │
│ ═════════════════════════════ │
│ │
│ Customer VPC / host │ Phony Cloud │
│ ┌──────────────┐ ┌──────────────┐ │ ┌────────────────┐ │
│ │ Database │◄────►│ Phony Agent │──────┼─►│ Control Plane │ │
│ │ (private) │ TLS/ │ │ out- │ │ (metadata only)│ │
│ │ │ socket │ bound│ └────────────────┘ │
│ └──────────────┘ └──────────────┘ only │ │
│ │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ OPTION: SSH tunnel (agent-local) │
│ ════════════════════════════════ │
│ When the agent runs in a different network segment than the DB │
│ (e.g. agent on a CI host, DB behind a bastion), the AGENT opens │
│ an SSH tunnel — still entirely within customer infrastructure. │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Database │◄───►│ Bastion │◄───►│ Phony Agent │ │
│ │ (private) │ │ (SSH key) │ SSH │ (customer CI │ │
│ └──────────────┘ └──────────────┘ │ or VM) │ │
│ └──────────────┘ │
│ │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ENTERPRISE: Air-gapped │
│ ═══════════════════════ │
│ Fully self-hosted control plane inside the customer network — │
│ the natural extension of the architecture. No outbound │
│ connection to Phony at all. See the technical architecture │
│ page (/spec/technical/architecture). │
│ │
└─────────────────────────────────────────────────────────────────────────┘There is no cloud-side VPC peering, IPSec VPN, or "allowlist Phony's IPs" — those exist only in architectures where the vendor's cloud must reach into the customer's database. Here it never does.
Connection Configuration
Connection details (including credentials) are configured on and stored by the agent, in customer infrastructure. The control plane knows a connection exists (name, engine, host label) but never holds credentials.
{
"source": {
"type": "postgresql",
"host": "db.internal.company.com",
"port": 5432,
"database": "production",
"credentials": {
"type": "local_secret",
"ref": "env://PHONY_SOURCE_DSN"
},
"tunnel": {
"type": "ssh",
"bastion_host": "bastion.internal.company.com",
"bastion_user": "phony-agent",
"private_key_path": "/etc/phony/keys/bastion"
},
"ssl": {
"mode": "verify-full",
"ca_cert_path": "/etc/phony/certs/ca.pem"
}
}
}Schema Analysis
Schema introspection runs in the agent, against the local database. The resulting schema map and column classifications (names, types, PII labels, confidences — never values) are reported to the control plane, where they power the PII inventory and compliance reports.
Automatic PII Detection
The Schema Analyzer uses multiple strategies to detect PII columns:
┌─────────────────────────────────────────────────────────────────────────┐
│ PII DETECTION PIPELINE (runs in the agent) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. COLUMN NAME HEURISTICS (Fast) │
│ ════════════════════════════════ │
│ Pattern matching on column names: │
│ • email, mail, e_mail → EMAIL type │
│ • phone, tel, mobile, gsm → PHONE type │
│ • first_name, fname, ad → NAME type │
│ • ssn, tc_kimlik, national_id → NATIONAL_ID type │
│ • address, adres, street → ADDRESS type │
│ • credit_card, cc_number → CREDIT_CARD type │
│ │
│ 2. DATA SAMPLING (Accurate) │
│ ════════════════════════════ │
│ Sample N rows LOCALLY and analyze content (samples never leave │
│ the agent): │
│ • Regex patterns (email format, phone format) │
│ • Statistical analysis (entropy, uniqueness) │
│ • Named entity recognition (names, addresses) │
│ │
│ 3. ML CLASSIFICATION (Enterprise) │
│ ══════════════════════════════════ │
│ Fine-tuned model, shipped to and executed by the agent: │
│ • Healthcare: MRN, diagnosis codes │
│ • Finance: account numbers, transactions │
│ • Custom training on customer data patterns (in customer infra) │
│ │
│ OUTPUT (this classification metadata is what goes to the cloud): │
│ ═══════════════════════════════════════════════════════════════ │
│ { │
│ "users.email": { "pii_type": "EMAIL", "confidence": 0.98 }, │
│ "users.phone": { "pii_type": "PHONE", "confidence": 0.95 }, │
│ "users.bio": { "pii_type": "FREE_TEXT", "confidence": 0.72 } │
│ } │
│ │
└─────────────────────────────────────────────────────────────────────────┘Foreign Key Detection
Preserving referential integrity is critical. FK detection also runs in the agent:
-- Automatic FK detection via:
-- 1. Database metadata (information_schema)
-- 2. Naming conventions (user_id, order_id)
-- 3. Data analysis (matching value distributions) — local only
-- Result: Dependency graph (reported to control plane as metadata)
orders.user_id → users.id
order_items.order_id → orders.id
payments.order_id → orders.idTransform Pipeline
The transform pipeline runs entirely inside the agent, in the customer's network: read from the source DB, anonymize/synthesize with the embedded Rust engine and locally trained models, write to the target DB. The control plane never sits in the data path.
Streaming Architecture
Large databases are processed in streams, never loaded fully into memory:
┌─────────────────────────────────────────────────────────────────────────┐
│ STREAMING TRANSFORM PIPELINE (inside the phony agent) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ SOURCE DB PHONY AGENT TARGET DB │
│ ═════════ ═══════════ ═════════ │
│ │
│ ┌────────┐ ┌────────────────────────────────┐ ┌────────┐ │
│ │ │ │ │ │ │ │
│ │ users │───►│ ┌──────┐ ┌──────┐ ┌──────┐ │───►│ users │ │
│ │ 10M │ │ │ Read │ │Trans-│ │ Write│ │ │ 10M │ │
│ │ rows │ │ │Batch │─►│form │─►│Batch │ │ │ rows │ │
│ │ │ │ │ 10K │ │ │ │ 10K │ │ │ │ │
│ └────────┘ │ └──────┘ └──────┘ └──────┘ │ └────────┘ │
│ │ ▲ │ │ │
│ │ │ ┌──────────┐ │ │ │
│ │ └────│ Buffer │◄──┘ │ │
│ │ │ Queue │ │ │
│ │ │ (memory) │ │ │
│ │ └──────────┘ │ │
│ │ │ │
│ │ Parallel workers: 4-16 │ │
│ │ Batch size: 1K-100K │ │
│ │ Memory limit: 512MB-4GB │ │
│ │ │ │
│ └───────────────┬────────────────┘ │
│ │ progress + row counts (no rows) │
│ ▼ │
│ Control plane (job status) │
│ │
└─────────────────────────────────────────────────────────────────────────┘Transform Types
Transform rules are authored in the dashboard (or CLI) and delivered to the agent as configuration; the agent applies them locally.
{
"tables": {
"users": {
"columns": {
"id": { "transform": "KEEP" },
"email": {
"transform": "ANONYMIZE",
"generator": "template",
"pattern": "{{uuid}}@example.com"
},
"first_name": {
"transform": "ANONYMIZE",
"generator": "model",
"source": "tr_TR/first_names"
},
"phone": {
"transform": "MASK",
"pattern": "+90 5** *** **{last2}"
},
"password_hash": { "transform": "KEEP" },
"created_at": { "transform": "KEEP" },
"notes": { "transform": "NULL" }
}
}
}
}Sync Modes
All modes execute in the data plane; the control plane schedules them and records outcomes.
Full Sync
Complete table copy with transformation:
┌─────────────────────────────────────────────────────────────────────────┐
│ FULL SYNC PROCESS │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. SCHEMA SYNC │
│ • Drop existing tables (if exists) │
│ • Create tables with same schema │
│ • Create indexes after data load │
│ │
│ 2. DATA SYNC (per table, ordered by FK dependencies) │
│ • Disable FK constraints │
│ • Truncate target table │
│ • Stream + Transform + Bulk insert │
│ • Enable FK constraints │
│ │
│ 3. POST-SYNC │
│ • Rebuild indexes │
│ • Update statistics │
│ • Verify row counts │
│ • Report job summary to control plane (counts, duration) │
│ │
│ Duration: ~1 hour per 100GB (depends on transforms) │
│ │
└─────────────────────────────────────────────────────────────────────────┘Incremental Sync
Only sync changes since last sync:
┌─────────────────────────────────────────────────────────────────────────┐
│ INCREMENTAL SYNC STRATEGIES │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ STRATEGY 1: Timestamp-based (Simple) │
│ ══════════════════════════════════════ │
│ Requires: updated_at column on tables │
│ │
│ SELECT * FROM users │
│ WHERE updated_at > '2024-01-15 10:00:00' │
│ │
│ Pros: Simple, works everywhere │
│ Cons: Misses deletes, clock skew issues │
│ │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ STRATEGY 2: CDC (Change Data Capture) - Recommended │
│ ═════════════════════════════════════════════════════ │
│ Uses database replication log (the agent is a local replication │
│ client — the log never leaves the network): │
│ • PostgreSQL: Logical replication slots │
│ • MySQL: Binary log (binlog) │
│ │
│ Captures: INSERT, UPDATE, DELETE │
│ Latency: Near real-time (seconds) │
│ │
│ Pros: Complete, no missed changes │
│ Cons: Requires DB config, more complex │
│ │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ STRATEGY 3: Soft Delete Tracking │
│ ═════════════════════════════════ │
│ Requires: deleted_at column + no hard deletes │
│ │
│ Combined with timestamp-based for complete picture │
│ │
└─────────────────────────────────────────────────────────────────────────┘Subset Sync
Sync a representative subset for development:
{
"subset": {
"strategy": "percentage",
"percentage": 1,
"preserve_integrity": true,
"seed_tables": ["users"],
"rules": {
"users": {
"sample": "1%",
"filter": "created_at > '2024-01-01'"
},
"orders": {
"sample": "follow_fk",
"parent": "users"
},
"order_items": {
"sample": "follow_fk",
"parent": "orders"
}
}
}
}What the Control Plane Sees — and Never Sees
The privacy contract of the architecture, stated explicitly:
| The control plane sees (metadata) | The control plane never sees |
|---|---|
| Schema structure: table/column names, types, row counts | Row data — not in transit, not at rest, not in logs |
| Column classifications: PII type + confidence per column | Sampled values used for PII detection (analyzed locally, discarded) |
| Transform configuration (which rule applies to which column) | Database credentials (stored by the agent, in your infra) |
| Job state: scheduled/running/succeeded/failed, durations, row counts | Trained model contents (models live in your infra; only name + hash reported) |
| Report artifacts: PII inventory, KVKK/GDPR report packs | Snapshot contents (stored in your storage; see Snapshots) |
| Audit events: who ran what, when | Query results of any kind |
Everything in the left column is what the paid product is made of — orchestration, visibility, and the compliance artifact. Everything in the right column stays inside the customer's network by construction, not by policy.
Consistency Guarantees
Referential Integrity
Foreign keys are preserved through deterministic ID mapping:
┌─────────────────────────────────────────────────────────────────────────┐
│ FK CONSISTENCY │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ SOURCE: TARGET: │
│ ════════ ═══════ │
│ │
│ users users │
│ ┌─────┬───────────┐ ┌─────┬───────────┐ │
│ │ id │ email │ │ id │ email │ │
│ ├─────┼───────────┤ Transform ├─────┼───────────┤ │
│ │ 1 │ a@x.com │ ──────────► │ 1 │ xx@ex.com │ │
│ │ 2 │ b@x.com │ │ 2 │ yy@ex.com │ │
│ └─────┴───────────┘ └─────┴───────────┘ │
│ │
│ orders orders │
│ ┌─────┬─────────┐ ┌─────┬─────────┐ │
│ │ id │ user_id │ │ id │ user_id │ │
│ ├─────┼─────────┤ ID preserved ├─────┼─────────┤ │
│ │ 101 │ 1 │ ──────────────►│ 101 │ 1 │ ✓ FK intact │
│ │ 102 │ 2 │ │ 102 │ 2 │ │
│ └─────┴─────────┘ └─────┴─────────┘ │
│ │
│ KEY INSIGHT: IDs are NEVER transformed, only PII columns │
│ │
└─────────────────────────────────────────────────────────────────────────┘Transaction Consistency
Sync jobs are atomic at the table level:
- If a table sync fails, it's rolled back
- Other tables remain consistent
- Retry mechanism with exponential backoff (agent-local; the control plane just records attempts)
- Persistent failures surface as failed jobs in the dashboard with agent-side error context
Performance
Throughput is bounded by the customer's own hardware and network — the agent runs where the data is, so there is no WAN hop in the data path.
| Database Size | Full Sync | Incremental | Subset (1%) |
|---|---|---|---|
| 1 GB | ~2 min | ~10 sec | ~5 sec |
| 10 GB | ~15 min | ~30 sec | ~20 sec |
| 100 GB | ~2 hours | ~2 min | ~2 min |
| 1 TB | ~20 hours | ~10 min | ~10 min |
Optimization levers:
- Parallel workers (up to 16)
- Batch size tuning
- Index-free loading (rebuild after)
- Connection pooling
Scheduling
Schedules are defined in the control plane; the agent polls (outbound-only) and executes. If the control plane is unreachable, the agent can still be driven directly via the CLI.
{
"schedule": {
"type": "cron",
"expression": "0 2 * * *",
"timezone": "Europe/Istanbul",
"mode": "incremental",
"fallback_to_full": {
"enabled": true,
"after_days": 7
}
},
"notifications": {
"on_success": ["slack://channel"],
"on_failure": ["email://team@company.com", "pagerduty://service"]
}
}Notifications are dispatched by the control plane from job-status metadata.