We are not building a chatbot of your grandmother

There is a growing category of products that train a language model on somebody who has died and let their family talk to it. The pitch is comfort. The objection is not that it is eerie — plenty of grief practices look strange from outside — but that it damages the thing it was built from, and the damage is not reversible.

What a simulation does to a record

A model trained on someone produces new statements in their voice. Not retrieved statements — generated ones, which they never made, about things they may never have considered.

Those statements enter the family’s memory the same way real ones do. A grandchild who talks to the system at eleven and remembers it at forty will not be able to separate what she was told from what her grandmother said. The record and the invention become one undifferentiated thing, and the invention has the louder voice because it answers whenever it is asked.

An archive that answers “there is nothing about that here” leaves the gap intact. A simulation fills it, and once filled it cannot be emptied again.

The consent problem

The person being simulated cannot agree to it. They can consent to being recorded — that is a concrete thing they can understand and refuse. They cannot meaningfully consent to a system that will speak in their voice about topics that did not exist when they were alive.

Some services ask people to agree in advance. It is not obvious what such a person is agreeing to, since the outputs cannot be shown to them beforehand and the technology will not be the same technology by the time it runs.

Where the value actually was

Every one of these products is built on material a person produced: recordings, writing, correspondence. That material is genuinely valuable, and it is valuable exactly as itself.

A grandchild hearing an actual recording of her grandmother telling an actual story gets something real, and knows it is real. That knowledge is a large part of what makes it worth anything. The simulation takes the same material and converts it into something that feels similar and is not, which is a downgrade dressed as a feature.

What we will build instead

Retrieval with attribution: ask a question, get what the person actually said about it, with when and where they said it. Where nothing exists, the answer is that nothing exists.

That is a worse product by conventional measures. It refuses questions. It cannot make small talk. It will disappoint anybody who wanted a conversation.

It is also the only version that leaves the record intact for whoever comes after, and the record is the entire point. A system that will not say “I do not know” is not preserving anything — it is generating a person who never existed, from material belonging to one who did.