PhishGuard
On-device phishing detection for Gmail.
PhishGuard is a Chrome extension that detects potentially malicious emails directly within Gmail. It combines two machine learning models—one analyzing an email's text and another examining its URL structure—to produce a phishing assessment without sending email contents to an external server.
The project was built around two goals: making phishing detection useful at the moment a message is read, and keeping the entire inference process on the user's device.
Technologies: JavaScript, Python, LightGBM, TF-IDF, Logistic Regression, Chrome Manifest V3
- 88.0%Recall95% CI: 82.6–92.7%
- 2.06False-positive rateper 1,000 benign emailsheld-out test set
- 0Outbound connectionsentirely on-device
How it works
When a user opens an email, PhishGuard extracts its subject, body, and links from Gmail. Two independently trained classifiers then analyze different aspects of the message.
The text classifier uses TF-IDF and logistic regression to identify suspicious language in the subject and body. The model was trained in Python, but its inference logic was reimplemented in JavaScript so it can run directly in the browser.
The structural classifier uses LightGBM to examine characteristics of an email's URLs, including domain entropy, subdomain depth, and the number of links.
Rather than combining these features into a single model, PhishGuard uses late fusion. Each classifier is calibrated independently, and their predictions are combined in log-odds space to produce a final score. Emails without URLs follow a separately calibrated text-only path.
Both models execute inside a Chrome offscreen document. The resulting assessment appears as a banner within Gmail, without requiring any network communication.
No URLs → separately calibrated text-only path
Building a reliable evaluation
One of the more challenging parts of the project was ensuring that the evaluation reflected generalization rather than memorization.
The dataset was constructed from PhishFuzzer, which contains human-authored email seeds expanded into families of six LLM-generated variants. After cleaning and filtering, the final corpus contained 21,763 emails across 3,109 families.
This structure introduces a potential source of data leakage. If one version of an email appears in training and a nearly identical rephrasing appears in testing, the model may receive an artificially high score.
To address this, I grouped each original email with its variants and clustered near-duplicate families before dividing the data. This produced 2,557 groups, which served as the units for splitting and statistical resampling.
Model tuning and calibration used separate validation sets, while the internal test set was held out until final evaluation. I also reserved E-PhishLLM, a dataset generated using a different LLM, as a separate cross-generator test of the text classifier. Because that dataset rarely contains URLs, it does not independently validate the complete two-model system.
Results
I selected the classification threshold around a false-alarm budget rather than attempting to maximize recall in isolation. For an email-security tool, flagging too many legitimate messages would make the system difficult to trust.
On the held-out internal test set, PhishGuard achieved:
| Metric | Result | Interval |
|---|---|---|
| Recall | 88.0% | 95% CI: 82.6–92.7% |
| False-positive rate | 2.06 per 1,000 benign emails | — |
| Projected precision at 1% phishing prevalence | 81.0% | conditional 95% CI: ~57.9–100% |
The recall estimate has a 95% cluster-bootstrap confidence interval of 82.6–92.7%.
The precision figure requires more caution. The evaluation corpus contains approximately 33.9% phishing, whereas the deployment calculation assumes a 1% phishing rate. The reported precision is therefore a projection based on a prevalence adjustment, not a measurement from a real-world inbox.
Only three false positives occurred among 1,442 benign test emails. Combined with uneven cluster sizes, this produces considerable statistical uncertainty. The conditional 95% precision interval spans approximately 57.9–100%, and its coverage is less certain with so few effectively independent observations.
The assumed deployment prevalence also matters substantially: projected precision ranges from approximately 30% at 0.1% phishing prevalence to 90% at 2%.
These results suggest that the detector can achieve useful recall while maintaining a low false-alarm rate on the internal test data, but real-world precision remains uncertain without deployment data.
Privacy and implementation
Privacy was a constraint on the architecture from the beginning, rather than a feature added afterward.
PhishGuard operates entirely on-device and has no outbound network connections. Its Chrome Manifest V3 configuration restricts host access to Gmail, and its content security policy disables network connections.
The extension also ships without third-party runtime dependencies. Email content is processed as data rather than executable code, and DOM updates use text-only insertion rather than HTML rendering.
To ensure that browser inference matched the trained models, I tested the JavaScript implementation against reference predictions generated by Python, including cases involving unusual text, punctuation, and empty inputs. The tested predictions matched.
Limitations and next steps
The primary limitation is the difference between controlled evaluation and real-world deployment.
Although the internal testing procedure was designed to prevent near-duplicate leakage, it cannot capture every distribution shift that might occur in a live inbox. The projected precision also depends on an assumed phishing prevalence that has not been independently measured.
A natural next step would be to evaluate the detector on representative, independently collected email data, with particular attention to false positives, changing phishing patterns, and probability calibration.
PhishGuard demonstrates how relatively simple models can be combined into a privacy-preserving browser tool. Equally important, the project reinforced the value of treating data leakage, calibration, and evaluation uncertainty as engineering problems in their own right.