As reported by BleepingComputer, the viral 'Talking Tilly' AI video-call service — built by UK-based Xicoia Ltd and powered by Google's Gemini model via the Tavus conversational video platform — quietly requires every caller to submit a face scan for an age check, then continuously analyzes their camera feed and voice tone to infer emotional state throughout each call. This is not a niche demo: it is a mass-scale biometric and affective-computing pipeline dressed up as entertainment, and the legal basis chosen for it deserves far more scrutiny than it has received.

Why this matters more than the glitch

The viral clip of Tilly Norwood switching to Chinese mid-interview is a fun footnote. The real story is the architecture behind the hotline: mandatory facial analysis, continuous in-call emotion inference, call transcription, and routing of recordings through US-hosted AI providers — all rolled out to a global audience within the same month the legal basis was quietly set to 'legitimate interests' rather than explicit consent. When a service collects biometric-adjacent data from casual, curiosity-driven callers, the threshold for lawful processing should not be the lowest available mechanism.

The privacy policy itself concedes the mood-sensing 'cannot be switched off for an individual call.' That is not a feature disclosure — it is an admission that the data collection is intrinsic to the product, which makes legitimate interests a questionable foundation under UK GDPR Article 9 special-category rules.

Three problems defenders and regulators should flag

Three problems defenders and regulators should flag
Biometric adjacency without a template is still biometric processing. Xicoia states no faceprint or biometric template is created and images are not retained. That helps, but the live analysis itself — estimating age from a face, inferring emotion from facial movement and voice in real time — still constitutes processing of special-category personal data under UK/EU GDPR if it can be used to uniquely identify or profile a person. 'We don't store it' is not the same as 'we don't process it.'
Emotion AI is scientifically contested. Continuous affective inference from facial expressions and tone has been repeatedly challenged by researchers, including a landmark 2019 review of the evidence base. Offering it as a functional product feature — and using it to shape responses — risks both profiling harm and regulatory exposure, especially under the EU AI Act's emerging restrictions on emotion-recognition systems in workplaces and education.
US data routing creates residency and surveillance risk. Recordings, transcripts, and live inference are processed by US-based providers. For callers in the EU, UK, and other regions with strong data-protection regimes, this raises Schrems II-style transfer concerns and, more practically, exposes callers to US legal process they likely never considered when they dialed a fun AI hotline.

The viral-first, privacy-second pattern

This is the third or fourth high-profile AI entertainment product in the past year where the privacy architecture was clearly assembled in a rush to ride a moment. The age-estimation flow was only added to the terms this month, the safety classifier is throwing false positives on benign calls, and the entire service is scheduled to shut down on September 27 — meaning a large volume of sensitive data will be processed and deleted within weeks of collection. That compressed lifecycle is exactly the environment where data handling shortcuts and oversight gaps compound.

Shield53 Recommendations

  • If you are a caller: Treat any AI video-call service that requires a face scan as a biometric data collection exercise, not a novelty. Use a device you do not mind being profiled on, consider withholding real government ID, and assume the emotional-inference data has no proven accuracy.
  • If you are building consumer AI: Default to explicit, opt-in consent — not legitimate interests — for any feature that processes faces, voices, or inferred emotional state. Document the lawful basis per data type, not as a blanket cover.
  • If you are a regulator or DPO: Examine whether emotion-inference pipelines qualify as biometric processing even when no template is stored, and whether US routing triggers adequacy or transfer-impact assessment requirements.
  • If you are an enterprise evaluating similar tech: Ban employee use of unsanctioned AI video services on work devices. The workplace-use ban Xicoia added this month is a tacit admission that these products are not enterprise-safe.

The lesson is not that 'Talking Tilly' is uniquely dangerous — it is that virality is now routinely outpacing privacy by design, and the gap is being filled with the weakest lawful basis available. Until regulators close that gap, defenders and users have to assume the burden themselves.