With the nose and mouth covered by a mask, standard face recognition models get it wrong. This was a provincial undergraduate innovation project I led from 2021 to 2023, and it became my first paper.
The idea
FaceNet, a widely used face recognition model, is trained with a triplet loss: take one face as the “anchor”, pull another photo of the same person (the positive) closer, and push someone else’s face (the negative) away. It’s accurate on ordinary faces, but not on masked ones.
My change was to add one more point:
So I turned FaceNet’s triplet into a quadruplet, and the extra point is the masked face.
The training goal: the same person, masked or not, is pulled together; different people are pushed apart.
Data
Real masked photos are hard to find, so I used the open-source tool MaskTheFace to “put masks on” public face datasets, randomly applying blue and green surgical masks, N95 and KN95.
- Training: CASIA-WebFace (over 10,000 people, nearly 500,000 images) and its masked version;
- Testing: MFR2 with real masked faces, LFW with ordinary faces, and LFW-M, where half of LFW’s photos get a random mask.
The training pipeline was built on FaceX-Zoo, a PyTorch face recognition toolkit, with ResNet-50 as the backbone.
Results
Trained on CASIA-WebFace and compared with two baselines (accuracy, %):
| Test set | Quadruplet (mine) | Triplet + masked data | Triplet |
|---|---|---|---|
| LFW-M (half masked) | 95.73 | 95.68 | 90.80 |
| MFR2 (real masks) | 99.03 | 98.14 | 97.38 |
| LFW (no masks) | 96.63 | 95.60 | 96.01 |
It beat both baselines on all three test sets. The last row mattered most to me: learning masked faces didn’t cost it its ability to recognise ordinary ones.
The paper also notes its limits: trained on CelebA instead, it did worse than the baselines, possibly because that dataset is about half the size; and 96% on LFW is still short of the best results at the time, around 99%.
Process
I remember being in the lab during finals week, still tuning the model. The first thing I did every day was check the results, and if I wasn’t happy, adjust the parameters and run it again (with all sorts of problems along the way, like the RTX 3090 overheating and rebooting the machine).
The experiments were done, but the writing stalled for a long time. What finally worked:
Silence my inner judge, write down everything I wanted to say, then hand it to ChatGPT to polish.
The paper, Masked Face Recognition with Quadruplet Loss, was published at the 2023 IEEE International Conference on Image Processing and Computer Applications (ICIPCA 2023), with me as first author and Professor You Congzhe as supervisor.
Looking back
This helped me a lot later. When I worked with embeddings and vector search, “pull similar things together, push different things apart” was something I had already tuned by hand while training a face model.