Chrome MV3 · Gmail phishing detection · on-device
PhishGuard v1
A browser extension that flags phishing in the Gmail reading pane, on the device. Two detectors combine through calibrated late fusion inside an offscreen document: a TF-IDF + logistic-regression text head and a LightGBM structured head. No email leaves the machine.
The read-time pipeline
one email, end to end// an opened message, traced through the extension
The corpus, and how it was cleaned
every count traced to one measured frameThe training corpus is PhishFuzzer (human-authored phishing seeds, each expanded into a family of six LLM-rephrased variants).E-PhishLLM, a separate corpus from a different LLM generator, is held out as a cross-generator probe of the text head: since only ~4% of its emails carry a URL, it is scored entirely by the text-only head, with the structured head and router left out. It tests whether the text model generalizes to phishing from another generator, not the fused two-head system.
| Cleaning step | Seeds | Dropped |
|---|---|---|
| raw seed — feedstock | 3,300 | |
| empty body after trim | 3,297 | −3 |
| non-English filter | 3,192 | −105 |
| exact-dup on (subject, body) | 3,118 | −74 |
| LLM prompt-residue rows removed — generation prompt left in body | 3,116 | −2 |
| degenerate short bodies (hand-verified) | 3,109 | −7 |
final: 3,109 seeds · 18,654 variants · 21,763 rows
Family cascade
Seeds and variants were cleaned separately, so a dropped seed could leave its six variants behind. The cascade removes any variant whose seed did not survive, matched on Original_ID.
Result — 3,109 whole families, 0 orphaned, 0 malformed.
Text normalization
Mojibake — text corrupted by decoding UTF-8 through the wrong codec, leaving à  †artifacts — is repaired with ftfy before any other step. No mojibake signature remains in the cleaned set; legitimate accented text is preserved.
Zero-width, BOM, and bidirectional control characters are stripped, removing a source-marking artifact that would otherwise correlate with the label.
Leakage defenses
Every email address in subject and body is replaced with the token <EMAIL>, removing address strings that a model could use as a shortcut. The replacement is identical across phishing and benign mail, so it cannot introduce a new label signal.
Train/test splitting groups by Original_ID so a seed and its variants never separate, then applies complete-linkage clustering at τ=0.4 to keep cross-family near-duplicates — near-twins under different Original_IDs — on the same side of the split. Clustering finds 263 near-duplicate groups, reduced to 260 after merging three same-source campaigns it had split apart (§02).
How the data was separated
grouped so no near-twin can leakBecause each seed carries six near-identical variants, and some seeds are near-duplicates of other seeds, a naïve row-level split would let a near-twin sit in test while its twin trains — inflating every score. The split is built on groups, not rows: whole families stay together, and near-duplicate families are clustered together, so no near-twin ever straddles the train/test line.
The grouping
The 3,109 families resolve into 2,557 clusters: 260 hold two or more near-duplicate families (263 before a co-grouping check caught three same-source spam campaigns that complete-linkage had split across clusters, and merged them). The remaining 2,297 are standalone families with no near-twin. The split and the confidence intervals both treat every cluster — grouped or standalone — as one atomic unit.
Four folds, by group
- Train — 15,232 rows
- Validate (tune) — 2,177 rows
- Validate (calibrate) — 2,177 rows
- Internal test — 2,177 rows · 243 clusters · touched once
Variants inherit their seed's fold, so a family is never split across folds.
The threshold is stable
Sweeping τ shows the split is stable to small moves: at τ = 0.35 or 0.45, only ~1.5% of seeds (44–47 of 3,109) change grouping. Pushing higher progressively sheds near-duplicate pairs — 99 seeds regroup at τ = 0.5, 242 at τ = 0.6 — which would relax leakage protection, consistent with 0.4 being deliberately set at the conservative, near-duplicate-catching end.
Same-source campaign merge
Three legitimate campaigns — an Amex, an Adorama, and a bug-tracker thread — had fractured across adjacent clusters. Because they share a sender and template, they are genuinely dependent, so they were merged into one cluster each. This tightens the "clusters are independent" assumption the confidence intervals rest on.
Structured head — LightGBM
reads URL shapeA gradient-boosted decision tree (LightGBM) scores structural features of the message's URLs, then a fitted Platt sigmoid maps the raw score to a probability. Platt scaling fits a one-parameter logistic to the model's output so the number reads as a probability rather than an uncalibrated score.
The features
- URL domain entropy — how varied and random-looking the URL host's characters are (Shannon entropy), taken as the maximum over the message's domain hosts. High entropy indicates algorithmically generated or obfuscated domains. IP-literal hosts are excluded; the value is
0when the message has no URL. - Subdomain count — number of host labels minus two (
example.com= 0,thirdLabel.example.com= 1), maximum over domain hosts. Deep nesting such assecure.login.account.example.comis a common lookalike tactic. - URL count and presence — how many links the message carries and whether it has any. Presence also routes the message: mail with URLs goes to full fusion, mail without to the text-only head (§04).
Text head — TF-IDF + LogReg
reads the proseThe text head is a TF-IDF vectorizer feeding a logistic-regression classifier, reimplemented in pure JavaScript (text_model.js) so it runs in the offscreen document with no ONNX or WASM dependency. It reads the subject and body and outputs one probability. TF-IDF(term frequency–inverse document frequency) weights each word by how often it appears in the message against how rare it is across the corpus, so distinctive words count for more than common ones.
Pipeline
- Tokenize — split into runs of two or more Unicode word characters, lowercased, with accents folded first.
- Vectorize — count word unigrams and bigrams, weight by TF-IDF, normalize the vector to unit length.
- Classify — logistic regression over that vector:
sigmoid(coef · x + intercept).
Ported for parity, not translated loosely
The model is trained once in Python. The JavaScript re-implements only its predict_proba inference logic: the same tokenizer rule, n-gram joining, IDF weights, and coefficients, all read unchanged from the exported model. To catch any drift, the exporter records sklearn's probability for a battery of strings, including edge cases (unicode, punctuation, casing, empty input), and a harness checks the JavaScript against each. None tested have diverged — the parity that lets a Python-trained model run in the browser.
Fusion & calibration
two probabilities, one verdictThe two heads combine in log-odds: each calibrated probability is turned into a log-odds score, weighted, and summed. The fused score is then Platt-recalibrated and prior-shifted from the 33.9% training rate to an assumed 1% deployment rate. The heads are trained and calibrated on a corpus that is 33.9% phishing; a real inbox is far lower, so without the shift every score would overstate risk. The shift is an additive move in log-odds that changes the base rate while preserving each email's relative evidence.
p_final = σ( w₀ + w₁·logit p_text + w₂·logit p_struct )Two routes, by URL presence
Mail with URLs is scored by both heads and fused with the three-term weight set (w₀,w₁,w₂). Mail with no URLs has no structured signal, so it is scored by the text head alone under a separate two-term set (w₀',w₁') — fit only on URL-less calibration rows, not the pool.
The text-only route is fit independently (its own intercept and weight). Its scores aren't comparable to the fused scores, so the loader requires each route to carry its own threshold.
What the banner shows
A possible-phishing header, which route produced the score, the score itself, and a note that the score assumes an estimated phishing rate and is provisional. The banner fires at most once per message and does not re-fire as Gmail redraws the pane.
Results
how much to trust each numberBoth headline numbers carry 95% confidence intervals from acluster bootstrap — resampling the 243 internal-test near-duplicate clusters as whole blocks, not individual emails, so near-twins are never counted as independent evidence.
Recall — measured, tight
0.880 [0.826, 0.927]
Recall is prevalence-invariant — it carries no assumption about how common phishing is. Estimated across 89 clusters that contain phishing, the interval is narrow: the model catches ~88% of phishing.
Precision — projected, wide
0.810 [0.579, 1.000]
Precision at an assumed 1% deployment rate is extrapolated, not measured: no test emails came from a 1%-phishing inbox (the corpus is 33.9% phishing).
It rests on 3 false positives among 1,442 benign emails, distributed across an effective ~33 independent clusters (a Herfindahl count of the uneven cluster sizes, versus 243 nominal — one cluster monopolizes 16% of the data). Limited false positives and so few effective clusters leave the interval wide, right-skewed, and reliant on an asymptotic coverage argument that need not hold at this scale. Read it as the order of precision and a lower bound near 0.58, not a precise range.
Modest sensitivity to the largest cluster
Only dropping the dominant 357-row campaign moves the estimate, by ~0.05; the rest leave it flat. Its influence is real but modest.
Security posture
client-side, no networkmail.google.comNo email can leave the device. The extension holds a single host permission, requests no network permission, and its content-security policy pins connect-src to 'none'. With no outbound channel, nothing can be transmitted. The extension ships no third-party code: both heads are pure JavaScript written for this project. There is nothing to pull from a CDN and no supply chain to compromise beyond Chrome itself. These properties are enforced before release. A test suite audits the manifest during development, and a network-capable configuration cannot pass that audit.