core

core contains the CloakDF class - a pseudonymisation/anonymisation engine for consistent cross-table key replacement and column masking, with Fernet-encrypted storage of mappings.

Setup & data

We use the Northwind dataset — a classic sample database with customers, orders, and employees tables that share keys across them.

import httpx
from pathlib import Path

base = "https://raw.githubusercontent.com/neo4j-contrib/northwind-neo4j/refs/heads/master/data"
files = ["customers.csv", "orders.csv", "employees.csv"]
dest = Path("../data")
dest.mkdir(exist_ok=True)

for f in files:
    (dest/f).write_bytes(httpx.get(f"{base}/{f}").content)
    print(f"✓ {f}")

✓ customers.csv
✓ orders.csv
✓ employees.csv

import pandas as pd
for f in files:
    df = pd.read_csv(dest/f, on_bad_lines='skip')
    print(f"\n{f}: {df.shape}")
    print(df.columns.tolist())
    display(df.head(2))

customers.csv: (67, 11)
['customerID', 'companyName', 'contactName', 'contactTitle', 'address', 'city', 'region', 'postalCode', 'country', 'phone', 'fax']

	customerID	companyName	contactName	contactTitle	address	city	region	postalCode	country	phone	fax
0	ALFKI	Alfreds Futterkiste	Maria Anders	Sales Representative	Obere Str. 57	Berlin	NaN	12209	Germany	030-0074321	030-0076545
1	ANATR	Ana Trujillo Emparedados y helados	Ana Trujillo	Owner	Avda. de la Constitución 2222	México D.F.	NaN	05021	Mexico	(5) 555-4729	(5) 555-3745

orders.csv: (654, 14)
['orderID', 'customerID', 'employeeID', 'orderDate', 'requiredDate', 'shippedDate', 'shipVia', 'freight', 'shipName', 'shipAddress', 'shipCity', 'shipRegion', 'shipPostalCode', 'shipCountry']

	orderID	customerID	employeeID	orderDate	requiredDate	shippedDate	shipVia	freight	shipName	shipAddress	shipCity	shipRegion	shipPostalCode	shipCountry
0	10248	VINET	5	1996-07-04 00:00:00.000	1996-08-01 00:00:00.000	1996-07-16 00:00:00.000	3	32.38	Vins et alcools Chevalier	59 rue de l'Abbaye	Reims	NaN	51100	France
1	10249	TOMSP	6	1996-07-05 00:00:00.000	1996-08-16 00:00:00.000	1996-07-10 00:00:00.000	1	11.61	Toms Spezialitäten	Luisenstr. 48	Münster	NaN	44087	Germany

employees.csv: (9, 18)
['employeeID', 'lastName', 'firstName', 'title', 'titleOfCourtesy', 'birthDate', 'hireDate', 'address', 'city', 'region', 'postalCode', 'country', 'homePhone', 'extension', 'photo', 'notes', 'reportsTo', 'photoPath']

	employeeID	lastName	firstName	title	titleOfCourtesy	birthDate	hireDate	address	city	region	postalCode	country	homePhone	extension	photo	notes	reportsTo	photoPath
0	1	Davolio	Nancy	Sales Representative	Ms.	1948-12-08 00:00:00.000	1992-05-01 00:00:00.000	507 - 20th Ave. E. Apt. 2A	Seattle	WA	98122	USA	(206) 555-9857	5467	0x151C2F00020000000D000E0014002100FFFFFFFF4269...	Education includes a BA in psychology from Col...	2.0	http://accweb/emmployees/davolio.bmp
1	2	Fuller	Andrew	Vice President, Sales	Dr.	1952-02-19 00:00:00.000	1992-08-14 00:00:00.000	908 W. Capital Way	Tacoma	WA	98401	USA	(206) 555-9482	3457	0x151C2F00020000000D000E0014002100FFFFFFFF4269...	Andrew received his BTS commercial in 1974 and...	NaN	http://accweb/emmployees/fuller.bmp

Table configuration

Each table is configured with key groups (id1, id2, …) and a list of columns to mask. Key groups ensure consistent pseudonymisation across tables — e.g. customerID maps to the same UUID in both customers and orders because both use group id1.

tables = {
    'customers': {
        'id1': 'customerID',
        'mask': ['contactName', 'address', 'phone', 'companyName', 'fax', 'city']
    },
    'orders': {
        'id1': 'customerID',
        'id2': 'employeeID',
        'mask': ['shipName', 'shipAddress']
    },
    'employees': {
        'id2': 'employeeID',
        'mask': ['firstName', 'lastName', 'address', 'homePhone', 'birthDate', 'notes']
    },
}

Design decisions

Why UUID4 for key pseudonymisation?

UUID4 is generated from cryptographically random bits, no timestamp, MAC address, or sequence baked in. The fake ID reveals nothing about the original. With 122 random bits, collision probability is astronomically small.

Shared vault for masked columns

Masked columns (names, addresses, etc.) use a flat shared dict across all tables. If "Maria Anders" appears in both customers and orders, it gets the same hex token. This preserves referential consistency without leaking the original value.

setdefault pattern

Both key_maps and vault use dict.setdefault() , a get-or-insert in one call. This ensures existing mappings are reused and new values are only generated for unseen originals:

km = self.key_maps.setdefault(group, {})  # create or retrieve inner dict
km.setdefault(val, str(uuid.uuid4()))     # only insert if missing

Encryption at rest with Fernet

The mappings (key_maps + vault) are the most sensitive artefact ; they’re the keys to de-anonymise everything. We use Fernet (AES-128-CBC + HMAC) from the cryptography library. The encryption key is never stored on the CloakDF instance — it’s passed in at save()/load() time, keeping the class stateless with respect to secrets.

CloakDF class

---

source

CloakDF

def CloakDF(
    tables
):

Pseudonymise and mask DataFrames consistently across related tables.

Round-trip test

Encode all three tables, save encrypted mappings, load from a fresh instance, decode, and verify we get the originals back.

import os

# 1. Generate key and store in env var
key = CloakDF.generate_key()
os.environ['CLOAKDF_KEY'] = key.decode()
loaded_key = CloakDF.load_key(env_var='CLOAKDF_KEY')
assert loaded_key == key
print("✓ Key round-tripped via env var")

# 2. Encode all tables
clk = CloakDF(tables)
originals, encoded = {}, {}
for name in tables:
    df = pd.read_csv(dest/f"{name}.csv", on_bad_lines='skip')
    for k, v in tables[name].items():
        if k.startswith('id'): df[v] = df[v].astype(str)
    originals[name] = df
    encoded[name] = clk.encode(name, originals[name])
    print(f"✓ Encoded {name}")

# 3. Save, load with env key, decode and compare
# --- Key loading options (uncomment one) 
# loaded_key = CloakDF.load_key(path=dest/"secret.key")
# loaded_key = CloakDF.load_key(env_var='CLOAKDF_KEY')
clk.save(dest/"mappings.enc", loaded_key)
clk2 = CloakDF.load(dest/"mappings.enc", loaded_key, tables)
for name in tables:
    decoded = clk2.decode(name, encoded[name])
    pd.testing.assert_frame_equal(decoded, originals[name])
    print(f"✓ Decoded {name} matches original")

✓ Key round-tripped via env var
✓ Encoded customers
✓ Encoded orders
✓ Encoded employees
✓ Decoded customers matches original
✓ Decoded orders matches original
✓ Decoded employees matches original

cols = ['employeeID', 'firstName', 'lastName', 'title', 'address', 'homePhone']
display(encoded['employees'][cols].head(3))
display(decoded[cols].head(3))

	employeeID	firstName	lastName	title	address	homePhone
0	d64fe22d-1661-4591-925d-444a16a2b05d	347dbd51390f	5b5ed8433ade	Sales Representative	e8973b766f93	cabbc198a844
1	5a2fb4ef-1517-4cd1-8f78-8b194a38a8d3	1d2f56ece163	f6369845f53e	Vice President, Sales	432fb00bda39	eb8a4817cb53
2	986a68a7-1539-4108-b132-76f69cded2a2	bc63cd2ad727	77df49f480ac	Sales Representative	2b9e99510ab1	d92bb310c745

	employeeID	firstName	lastName	title	address	homePhone
0	1	Nancy	Davolio	Sales Representative	507 - 20th Ave. E. Apt. 2A	(206) 555-9857
1	2	Andrew	Fuller	Vice President, Sales	908 W. Capital Way	(206) 555-9482
2	3	Janet	Leverling	Sales Representative	722 Moss Bay Blvd.	(206) 555-3412