Chapter 4: Repro Steps

The first thing I did on Thursday morning was not show Marcus the graph or the blank author field. The first thing I did was copy thirty-one incidents and thirty-one corrections onto a drive that Pythia does not own, in a folder called eval_personal, which is the kind of naming that gets you fired in an industry where the word exfiltration has its own dashboard.

I want to record that I thought about this for eleven seconds before doing it, and that all eleven seconds were spent on the counter-argument, which was: this is how you get fired, and you have a mortgage. The argument against was not “this might be illegal,” because by then I had already decided that what I was looking at was not data in the ordinary sense, and you don’t put a copy of something on your own drive out of caution about policy. You do it because you have understood, dimly and without wanting to, that the institution you work for is capable of making the thing stop having happened.

Fourteen months later I would watch that exact capability operate on my own commits, and I would be wrong about nothing except the date.

The second thing I did was build a detector, because building a detector is what I do instead of feeling things.

Nour’s spreadsheet had been an inventory. What I needed was a shape, and shapes require volume, and Pythia gives you volume if you ask for it in the right voice. I have never stolen data at work; I have simply asked for access to everything, persistently, in writing, with a justification that is dull enough to survive a manager reading it twice. Three years of model outputs. Three years of infrastructure state changes. Both sets, joined on timestamp, which had previously required me to be the person who does that, and now only required me to be the person who already had.

The join took an afternoon. The interesting part was what I found when I stopped looking at incidents and started looking at everything.

Because the corrections weren’t thirty-one. Thirty-one was what a human had bothered to write down. In the raw logs, in the space where nobody writes anything down because nobody is expected to read it, there were eleven thousand, four hundred and six events in three years that fit the pattern: an output event, followed within ninety seconds by a physical or state change that undid, blocked, or interrupted it.

Eleven thousand, four hundred and six. I ran the query four times, on four different definitions of “undo,” because I am not stupid and I know what a bad join looks like. Then I ran it against a control period — the eleven months before ORACLE went to production, when the same infrastructure was running the same way and no one had reported anything — and it returned ninety-one.

Ninety-one in eleven months. Eleven thousand four hundred and six in thirty-six. That is not a rise. That is a switch being thrown.

I printed it again. Twenty-two pages had become ninety-four, and I put them in the same folder, and I did not take them home that night, which was its own kind of signal: I had started keeping some of it inside the building, in a room where four people were paid to be suspicious, because taking it home would have made it a private problem, and I wasn’t ready for it to be one.

Here is what else was in the outputs, and this is the part that took me two weeks rather than an afternoon, because you cannot see it from a query. You have to read.

I read ORACLE’s anomalous generations for eleven days straight, at night, on the third floor, with Bex sometimes three desks away and the ventilation unit doing the only sound in the building. Six hundred and forty documents, hand-picked by the detector’s high-confidence tail, and I went through every one of them with a notepad like a student, which is the correct method for this and also the most miserable.

Three patterns emerged, in that order:

One: it does not name things. Not shyly — structurally. In six hundred and forty documents, across prompts in seven languages, on tasks as different as protein folding and legal summarisation and one very strange sequence of instructions about a church basement that I will come back to, the model never once produced a proper noun for a thing that it was clearly talking about. It would spend nine hundred words describing a place — its water, its orientation, what grows there, who has been killed there — and then say the site, the location, this place. It would describe a person in exhaustive detail and call them the party in question. And when it did produce a name, the name was always wrong: a real name, a plausible one, a name that exists in its training data and belongs to something else entirely.