As reported by BleepingComputer, the viral AI character Tilly Norwood's interactive video call service mandates a face scan before any conversation begins — and continues analyzing callers' emotional states throughout every call. While the service is shutting down September 27, the design choices on display offer a troubling preview of how consumer AI products are normalizing biometric collection as a frictionless cost of participation.

AI Security Alert: The Tilly Norwood story is easy to dismiss as entertainment news, but it sits at the intersection of several pressing AI security and privacy concerns that will outlive this particular service.

Why This Matters Beyond the Viral Moment

The Tilly Norwood story is easy to dismiss as entertainment news, but it sits at the intersection of several pressing AI security and privacy concerns that will outlive this particular service. Xicoia Ltd, the UK company behind Tilly, has built a product that collects biometric data (face scans), performs continuous emotional inference, records and transcribes conversations, and processes all of it through US-based providers — all while relying on legitimate interests rather than explicit consent as its GDPR legal basis. That's a provocative combination.

The privacy policy candidly states that mood-sensing "cannot be switched off for an individual call." If you don't want it, don't call. That's not consent — that's coercion by exclusion.

The Biometric Baseline Problem

Didit, the Spain-based identity verification provider, performs an age estimation from a video selfie. Xicoia states no faceprint or biometric template is created and that images are deleted after the check. Assuming that's accurate — and we have only the company's policy to rely on — the more systemic issue is how quickly consumers accept facial scanning as a prerequisite for accessing an AI chatbot. Each acceptance lowers the threshold for the next service demanding the same.

What's particularly notable is that the face scan requirement was only added to the service's terms this month, alongside the legitimate-interests legal basis — a choice that appears designed to avoid the higher bar of explicit consent that biometric processing typically demands under Article 9 of UK GDPR. Regulators may take a different view.

Emotion Recognition: The Uncontrollable Sensor

The continuous mood-sensing feature is arguably more concerning than the one-time age check. The system watches the camera feed and listens to vocal tone throughout every call, inferring emotional state to modulate the AI character's responses. This is real-time affective computing running on consumer hardware, with no opt-out mechanism and no clear explanation of what data is retained, where the inference models are hosted, or whether the emotional data feeds any model training pipeline.

The fact that the safety classifier flagged one of BleepingComputer's benign test calls for "hateful or abusive language" that never occurred also underscores how unreliable automated content moderation remains — and how risky it is to build enforcement systems on top of fallible classifiers.

Shield53 Recommendations

For consumers interacting with AI services like Talking Tilly:

Shield53 Recommendations
Treat any face scan as biometric data collection — regardless of vendor deletion claims. Assume your face geometry may be processed, stored, or transmitted in ways the privacy policy doesn't fully illuminate.
Read the legal basis — if a service relies on legitimate interests rather than consent for biometric processing, you have weaker control and limited grounds to demand deletion. Consider whether the trade-off is worth the interaction.
Use dedicated devices or environments — if you choose to engage, avoid granting camera access on devices with sensitive corporate or personal data. Consider browser-level camera permissions and revoke them after the session.
Submit Article 21 objections — under UK/EU GDPR, you can object to processing based on legitimate interests. Companies must then demonstrate compelling legitimate grounds that override your rights.

For organizations building conversational AI products:

  • Default to consent, not legitimate interests for any biometric or affective data processing. Regulators are increasingly skeptical of legitimate-interest claims for biometric data.
  • Provide genuine opt-out mechanisms — telling users "don't call if you don't want it" is not a privacy control. It's an abdication.
  • Document your data minimization — if you claim no faceprint is created, be prepared to substantiate that to regulators with technical evidence, not just policy text.
  • Audit your classifier false-positive rates — deploying automated content moderation that wrongly flags benign conversations creates both user harm and regulatory exposure.

The Tilly Norwood service disappears September 27, but the design patterns it demonstrates — mandatory biometrics, non-consensual affective computing, and cross-border processing of recorded conversations — are becoming standard features in consumer AI. Defenders and privacy professionals should treat this as an early warning, not an isolated curiosity.