# cloakdf


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Install

    pip install cloakdf

## How to use

We’ll demonstrate `cloakdf` using the [Northwind
dataset](https://github.com/neo4j-contrib/northwind-neo4j). A classic
sample database with customers, orders, and employees tables that share
keys across them.

``` python
#| eval: false
import httpx
from pathlib import Path

base = "https://raw.githubusercontent.com/neo4j-contrib/northwind-neo4j/refs/heads/master/data"
files = ["customers.csv", "orders.csv", "employees.csv"]
data = Path("../data")
data.mkdir(exist_ok=True)

for f in files:
    (data/f).write_bytes(httpx.get(f"{base}/{f}").content)
```

### 1. Define your table configuration

Each table’s config specifies two kinds of columns:

- **`id` groups** (e.g. `id1`, `id2`) — key columns that are shared
  across tables. Columns in the same group get **consistent
  pseudonyms**: `customerID` in both `customers` and `orders` maps to
  the same UUID, preserving referential integrity.

**Note:** All `id` columns must be string dtype before encoding — cast
with `.astype(str)` if needed. - **`mask`** — sensitive columns to
replace with opaque hex tokens. These are stored in a shared vault, so
identical values (e.g. the same address appearing twice) get the same
token.

``` python
tables = {
    'customers': {
        'id1': 'customerID',
        'mask': ['contactName', 'address', 'phone', 'companyName', 'fax', 'city']
    },
    'orders': {
        'id1': 'customerID',
        'id2': 'employeeID',
        'mask': ['shipName', 'shipAddress']
    },
    'employees': {
        'id2': 'employeeID',
        'mask': ['firstName', 'lastName', 'address', 'homePhone', 'birthDate', 'notes']
    },
}
```

### 2. Encode your DataFrames

``` python
import pandas as pd
from pathlib import Path

data = Path("../data")
unk = CloakDF(tables)

originals, encoded = {}, {}
for name in tables:
    df = pd.read_csv(data/f"{name}.csv", on_bad_lines='skip')
    for k, v in tables[name].items():
        if k.startswith('id'): df[v] = df[v].astype(str)
    originals[name] = df
    encoded[name] = unk.encode(name, df)

encoded['customers'].head(3)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }
&#10;    .dataframe tbody tr th {
        vertical-align: top;
    }
&#10;    .dataframe thead th {
        text-align: right;
    }
</style>

<table class="dataframe" data-quarto-postprocess="true" data-border="1">
<thead>
<tr style="text-align: right;">
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th">customerID</th>
<th data-quarto-table-cell-role="th">companyName</th>
<th data-quarto-table-cell-role="th">contactName</th>
<th data-quarto-table-cell-role="th">contactTitle</th>
<th data-quarto-table-cell-role="th">address</th>
<th data-quarto-table-cell-role="th">city</th>
<th data-quarto-table-cell-role="th">region</th>
<th data-quarto-table-cell-role="th">postalCode</th>
<th data-quarto-table-cell-role="th">country</th>
<th data-quarto-table-cell-role="th">phone</th>
<th data-quarto-table-cell-role="th">fax</th>
</tr>
</thead>
<tbody>
<tr>
<td data-quarto-table-cell-role="th">0</td>
<td>9eb2c3d4-32d3-434c-863c-44d8ac4067d9</td>
<td>9288db060f26</td>
<td>5c1dfdfc8f62</td>
<td>Sales Representative</td>
<td>80e2676376a1</td>
<td>c1f7bd6a22f0</td>
<td>NaN</td>
<td>12209</td>
<td>Germany</td>
<td>715e53fda07f</td>
<td>cd0efa7df34e</td>
</tr>
<tr>
<td data-quarto-table-cell-role="th">1</td>
<td>82d494d5-a9e7-4690-b5ec-3104dad5b87d</td>
<td>18db1aa91f8c</td>
<td>3c4420341c1a</td>
<td>Owner</td>
<td>135d66291b39</td>
<td>42d81129a68e</td>
<td>NaN</td>
<td>05021</td>
<td>Mexico</td>
<td>a5611db2c349</td>
<td>25fabc23caa4</td>
</tr>
<tr>
<td data-quarto-table-cell-role="th">2</td>
<td>fa465d8d-ead0-4d03-ae91-8a65715c7fd1</td>
<td>9458f0d21f98</td>
<td>08b3087d3dda</td>
<td>Owner</td>
<td>a93f1f586e03</td>
<td>42d81129a68e</td>
<td>NaN</td>
<td>05023</td>
<td>Mexico</td>
<td>2a648576b9be</td>
<td>NaN</td>
</tr>
</tbody>
</table>

</div>

### 3. Save encrypted mappings

The mapping dictionaries (`key_maps` and `vault`) are the sensitive
artefacts — they allow de-anonymisation. Generate a Fernet key and
encrypt them at rest. You can store the key to a file or an environment
variable:

> ⚠️ **Never commit your Fernet key or encrypted mappings to version
> control.** Ensure your `.gitignore` includes key files and `*.enc`
> files (e.g. `*.key`, `*.enc`, `data/`).

``` python
import os

key = CloakDF.generate_key()
unk.save(data/"mappings.enc", key)

# Optionally store the key in an environment variable
os.environ['CLOAKDF_KEY'] = key.decode()
print("✓ Mappings saved and key stored in env var")
```

    ✓ Mappings saved and key stored in env var

### 4. Load and decode

Load the encrypted mappings (using the key directly or from an
environment variable) and reverse the encoding:

``` python
# Load key from env var (or pass `key` directly)
loaded_key = CloakDF.load_key(env_var='CLOAKDF_KEY')
# loaded_key = CloakDF.load_key(path="path/to/keyfile")  # alternative: from file

unk2 = CloakDF.load(data/"mappings.enc", loaded_key, tables)

for name in tables:
    decoded = unk2.decode(name, encoded[name])
    pd.testing.assert_frame_equal(decoded, originals[name])
    print(f"✓ {name} round-trip OK")

decoded.head(3)
```

    ✓ customers round-trip OK
    ✓ orders round-trip OK
    ✓ employees round-trip OK

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }
&#10;    .dataframe tbody tr th {
        vertical-align: top;
    }
&#10;    .dataframe thead th {
        text-align: right;
    }
</style>

<table class="dataframe" data-quarto-postprocess="true" data-border="1">
<thead>
<tr style="text-align: right;">
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th">employeeID</th>
<th data-quarto-table-cell-role="th">lastName</th>
<th data-quarto-table-cell-role="th">firstName</th>
<th data-quarto-table-cell-role="th">title</th>
<th data-quarto-table-cell-role="th">titleOfCourtesy</th>
<th data-quarto-table-cell-role="th">birthDate</th>
<th data-quarto-table-cell-role="th">hireDate</th>
<th data-quarto-table-cell-role="th">address</th>
<th data-quarto-table-cell-role="th">city</th>
<th data-quarto-table-cell-role="th">region</th>
<th data-quarto-table-cell-role="th">postalCode</th>
<th data-quarto-table-cell-role="th">country</th>
<th data-quarto-table-cell-role="th">homePhone</th>
<th data-quarto-table-cell-role="th">extension</th>
<th data-quarto-table-cell-role="th">photo</th>
<th data-quarto-table-cell-role="th">notes</th>
<th data-quarto-table-cell-role="th">reportsTo</th>
<th data-quarto-table-cell-role="th">photoPath</th>
</tr>
</thead>
<tbody>
<tr>
<td data-quarto-table-cell-role="th">0</td>
<td>1</td>
<td>Davolio</td>
<td>Nancy</td>
<td>Sales Representative</td>
<td>Ms.</td>
<td>1948-12-08 00:00:00.000</td>
<td>1992-05-01 00:00:00.000</td>
<td>507 - 20th Ave. E. Apt. 2A</td>
<td>Seattle</td>
<td>WA</td>
<td>98122</td>
<td>USA</td>
<td>(206) 555-9857</td>
<td>5467</td>
<td>0x151C2F00020000000D000E0014002100FFFFFFFF4269...</td>
<td>Education includes a BA in psychology from Col...</td>
<td>2.0</td>
<td>http://accweb/emmployees/davolio.bmp</td>
</tr>
<tr>
<td data-quarto-table-cell-role="th">1</td>
<td>2</td>
<td>Fuller</td>
<td>Andrew</td>
<td>Vice President, Sales</td>
<td>Dr.</td>
<td>1952-02-19 00:00:00.000</td>
<td>1992-08-14 00:00:00.000</td>
<td>908 W. Capital Way</td>
<td>Tacoma</td>
<td>WA</td>
<td>98401</td>
<td>USA</td>
<td>(206) 555-9482</td>
<td>3457</td>
<td>0x151C2F00020000000D000E0014002100FFFFFFFF4269...</td>
<td>Andrew received his BTS commercial in 1974 and...</td>
<td>NaN</td>
<td>http://accweb/emmployees/fuller.bmp</td>
</tr>
<tr>
<td data-quarto-table-cell-role="th">2</td>
<td>3</td>
<td>Leverling</td>
<td>Janet</td>
<td>Sales Representative</td>
<td>Ms.</td>
<td>1963-08-30 00:00:00.000</td>
<td>1992-04-01 00:00:00.000</td>
<td>722 Moss Bay Blvd.</td>
<td>Kirkland</td>
<td>WA</td>
<td>98033</td>
<td>USA</td>
<td>(206) 555-3412</td>
<td>3355</td>
<td>0x151C2F00020000000D000E0014002100FFFFFFFF4269...</td>
<td>Janet has a BS degree in chemistry from Boston...</td>
<td>2.0</td>
<td>http://accweb/emmployees/leverling.bmp</td>
</tr>
</tbody>
</table>

</div>

### 5. Compare encoded vs decoded

Here’s what the employees table looks like — encoded (pseudonymised) vs
decoded (original):

``` python
cols = ['employeeID', 'firstName', 'lastName', 'title', 'address', 'homePhone']
display(encoded['employees'][cols].head(3))
display(decoded[cols].head(3))
```

### Text Redaction with `cloakdf.redact`

While
[`CloakDF`](https://giordafrancis.github.io/cloakdf/core.html#cloakdf)
handles **reversible** pseudonymisation of structured tabular data,
`redact` handles **irreversible** PII removal from free text. It’s
designed for case notes, narratives, and other unstructured fields where
there’s no mapping to reverse, you just need the sensitive information
gone.

It combines a distilbert NER model for detecting person names with regex
patterns for structured identifiers (emails, phone numbers, NHS numbers,
UK postcodes, and standalone numbers). Each detected entity is replaced
with a flat typed placeholder like `[NAME]`, `[EMAIL]`, `[PHONE]`, etc.,
so downstream consumers can still see *what kind* of information was
there without seeing the actual values.

The default model is
`Davlan/distilbert-base-multilingual-cased-ner-hrl`, a lightweight
multilingual NER model. In practice it captures around 98% of PII in
typical case-note text, but it is **not perfect** unusual name formats,
ambiguous identifiers, and non-English text can produce false negatives,
while common words occasionally get flagged as false positives. Always
review redacted output before sharing.

``` python
from cloakdf.redact import Redactor

r = Redactor()
sample = "Dr Jane Smith called from jane.smith@nhs.net on 07123 456789. NHS: 123 456 7890, postcode SW1A 1AA."
r.redact(sample)
```


    Loading weights:   0%|          | 0/102 [00:00<?, ?it/s]
    Loading weights: 100%|##########| 102/102 [00:00<00:00, 5849.88it/s]
    Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

    'Dr [NAME] called from [EMAIL] on [PHONE]. NHS: [NHS], postcode [POSTCODE].'