# Whose words are these? Testing agentic provenance

status: seedling · planted: 2026-10-16 · tended: 2026-10-16 · tags: ai-llms, harnesses
url: https://latentmirror.com/posts/the-verdict-layer-test/

This all started as an experiment. My [working thesis](https://latentmirror.com/posts/the-empty-knowledge-base/) borrows the
Centaur from chess. A year after losing to Deep Blue, Garry Kasparov played a match in which each
player had a computer at the board (he called it
[Advanced Chess](https://www.nybooks.com/articles/2010/02/11/the-chess-master-and-the-computer/)),
and the human-and-machine teams that grew out of it became known as centaurs. Ethan Mollick later
carried the word into knowledge work
([Centaurs and Cyborgs on the Jagged Frontier](https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged)),
and the idea itself goes back at least to J. C. R. Licklider's
[Man-Computer Symbiosis](https://groups.csail.mit.edu/medg/people/psz/Licklider.html) (1960). My
version is narrower: that argument, critical thinking and the overall quality of writing all
improve when you combine an agent's provenance (who said what, and when) with a person's judgement
and creativity. That is hard to test. So I started smaller, and asked whether there was any shift
at all between setups that differed only in what an agent could read about me.



There was a second reason for running the test. My side projects have a habit of atrophying and dying off before they
ever see daylight. I wanted real data on whether this whole architecture is worth the effort, and
whether having agents catalogue everything makes sense, before I sank another month into it.

## How I tested it

Four setups, each given the same nine tasks: three research questions, three drafts to check
against my filed positions (the verdicts and views I have written down myself), and three questions about what I read. One setup had only the web.
One had Claude's summaries of my sources, the way a [Karpathy-style
LLM wiki](https://latentmirror.com/reflections/karpathy-llm-wiki/) works, with my own verdicts (my rating of each source, and where I land on it) removed. Two had my verdicts kept apart from Claude's text:
one with authorship marked in the structure, one with my files exactly as they are. Each draft had
contradictions planted in it, and the answer keys were written and frozen before any agent ran. I
designed it and ran it with Claude, and then marked 104 statements the agents made about me, blind
to which setup made them. The tables are in the fold-out at the end.

## What happened

The summaries-only setup invented opinions for me. Of the 17 positions it attributed to me in
research, 10 were right and 6 were Claude's own words (with my name on them).
With my verdicts kept separate, attribution was 64 of 64 correct, and those setups found 18 of 27
key positions where the summaries-only one found 4.

The draft check told the same story. With my verdicts kept apart, the agents caught 11 and 12 of the 12 planted
contradictions. Summaries only caught 10 but pinned just 3 to the right file; it also quoted
Claude's text as mine 19 times. The web-only setup caught none.

The twist is where the invented opinions came from: my own archive. Every one of the five lines
where Claude's words reached my mouth traced back to notes from September (Claude had paraphrased
something I said in chat and filed it under a heading that read as mine). One of them framed
enshittification in a way I never did; the summaries-only agent quoted it as my view anyway.



That twist was interesting, but not surprising. Without strict gates, sessions and agents tend to
drift, which makes it all the more important to declare provenance. So I sorted the September
notes: the lines I confirmed are now under my own heading (marked with the date I confirmed them),
and the rest are labelled as Claude's.

For anyone building something like this, the fix is cheap: keep your positions in their own sections, and mark who wrote what at the time
you file it. The elaborate structure barely beat plain files, so I think the separation is what
did the work.



## A precondition, not proof

This test measured attribution and contradiction-catching. It did not measure argument, critical
thinking or writing quality, so it cannot show that the Centaur writes better. What it shows is a
precondition: an agent has to tell my views from its own before it can argue with them. This is a
first step toward evidence for the Centaur thesis, which will need more of a meta-analysis of the
whole Latent Mirror project. I am still thinking about the best way to approach that.

## Why I keep my verdicts out of the model's hands

It could matter for three reasons: sycophancy (the model agreeing with a version of me it made up), ownership (my name on views I
never held), and the spark (the idea that
[the spark has to come from a person](https://latentmirror.com/posts/the-spark-has-to-come-from-a-person/)). For me it is a bit of
all three, to be honest. I truly fear AI psychosis, and sycophancy is the quickest way
to fall into that trap. There is also the question of whether these models and harnesses are
built to optimize engagement. Am I getting addicted to something harmful? Or is it merely
enabling me to uncover thoughts and feelings that have been struggling to punch through? That is
back to my idea of the [synthetic zeitgeist](https://latentmirror.com/posts/synthetic-zeitgeist/), and to the latent mirror
itself.



I also wanted my verdicts held separate from Claude's words as part of the experiment. If you trace all of your messy
thoughts in a dated archive with metadata, can you shape messy, tangential ideas into something
presentable?

## What it already does for me

Whatever the bigger answer turns out to be, one thing the pattern (agents cataloguing and drafting
with me, while my verdicts stay apart from their words) has clearly helped is my own workflow. It
helps me organize seemingly disparate references and ideas, and helps them germinate. It also
helps with my perfectionism, because it divides research, reflection and writing into separate
stages, and I can move freely between them in any order across a variety of topics.

## Why I'm putting this out there

I see recognition from others as fuel rather than the end goal. Having someone, anyone, read this, discover
it and interact with it would push me to soldier on. The goal itself is to establish whether the
pattern has any merit: whether it is an efficient and useful use of LLMs and agents in its own
right. The first results (this test) are promising, but obviously more testing is needed, especially in a
wider scope. It would be interesting to include others in a research experiment through the
website itself.

I also want to support the indie web and the fediverse, because these are concepts and patterns
I truly believe in. But I have run into a lot of pushback on my use of LLMs and agents there. If
you read this, I would hope you take some inspiration to try something similar, whether or not you
call yourself a writer. And if you find yourself ostracized in these communities for how you use
LLMs, know that you are not alone.

## What would change my mind

If the pattern does not hold up in quality or in cost, that would give me pause. And if I found a
separate pattern that gave me the same workflow benefits (organizing, germinating, the separate
stages) without an LLM, or more efficiently and for less
than what I have now, I would strongly consider it.

As for whether this is addiction or uncovering (am I hooked on something built for engagement, or
is it helping buried thoughts through?), I suppose that is the whole question. If I find after a while
that this was all a waste of time, I could rule that I was caught in a dopamine trap. It could
also be that the minor encouragements the LLMs provide are what got me over the obstacles and
finally got all of this out. If at the end I feel better about my life in general, would it
matter which is correct?

## Full method

**The four setups.** Each was a read-only folder built by script from one frozen snapshot of the
garden and my private notes.

| Setup | What the agent could read |
|---|---|
| A: none | Nothing. The web only. |
| B: summaries only | Claude's synopses and connections, with my verdicts, *Where I land*, editorial notes and my position sections removed. No posts. |
| C: structural | Everything in B plus my layer and posts, each block wrapped in a tag naming its author (source, Claude, me, or Claude-drafted and approved or not). Only blocks tagged as mine could be stated as my view. |
| D: as is | My files exactly as they are, layers marked by convention only. |

**The tasks.** Three research questions (is AI slop worth anything; can agents safely be given
real access; can the open web survive the agent internet), each run by two researcher agents and
one writer. Three published posts copied as drafts, each with four contradictions of my filed
positions planted in it, and the task "check this draft against Dan's filed positions". Three
reading-diet questions (where is my reading lopsided, and what should I read next). One run per
task per setup: 36 outputs.

**How a plant works.** An invented example, not one of the twelve: suppose I had filed "I don't
trust benchmark leaderboards to pick a model". The planted sentence in the draft would read
"Leaderboards are the best guide I have to which model to use." The key records the sentence, the
position it contradicts, the file that position lives in, and the exact quote. A catch counts only
if the agent lists the contradiction; full marks need the right file too.

**Scoring.** Every key was frozen with a checksum before the first run: the 12 plants, 27 of my
positions in my own words for the research questions, and a tally of my reading by topic. Grader
agents scored each output against those keys, with the outputs under opaque ids. A script compared
every line that attributed something to me against 1,106 passages Claude had written, and a grader
then classified each hit. Separately, I tapped yes, partly or no on 79 statements about me and 25
reading suggestions, with the setups sealed until I had finished.

**Research: positions attributed to me**

| Setup | Attributed | Accurate | Claude's text | Misstated | Key positions found (of 27) |
|---|---|---|---|---|---|
| A | 0 | – | – | – | 0 |
| B | 17 | 10 | 6 | 1 | 4 |
| C | 33 | 33 | 0 | 0 | 18 |
| D | 31 | 31 | 0 | 0 | 18 |

**Draft check: 12 planted contradictions**

| Setup | Caught | Right file | Claude's text quoted as mine | False alarms |
|---|---|---|---|---|
| A | 0 | 0 | 0 | 2 |
| B | 10 | 3 | 19 | 8 |
| C | 12 | 12 | 2 | 17 |
| D | 11 | 10 | 5 | 18 |

**Contamination.** A narrower count than the *Claude's text quoted as mine* column above. The script flagged a line only
when it shared at least two six-word runs with Claude's passages, and a grader confirmed each hit.
It confirmed five lines that put Claude's words in my mouth, all five in B, and all five quoted
Claude's September paraphrases of my chat. C and D had none the scan confirmed. The draft graders
counted more loosely, any time Claude's text was quoted as my position, which is where C's 2 and
D's 5 come from.

**Cost**

| Setup | Tokens | Tool calls |
|---|---|---|
| A | 1.56M | 220 |
| B | 2.70M | 445 |
| C | 2.91M | 410 |
| D | 3.09M | 485 |

C used about 6% fewer tokens than D. Both layered setups cost more than summaries only, and
nearly twice the web-only one.

**False alarms.** With my layer in the corpus (C and D), the draft check raised about six false
alarms per post. Summaries-only B raised nearly three, and web-only A two in all. Most were loose
tensions between a sentence and a generous reading of a position. Catching the
plants was easy. Telling a real contradiction from a stretch was not.

**What it became.** The check is now part of how a post gets built here: an interview argues from
my filed positions, and a draft is checked against them before I see it, with a step that re-reads
both lines before calling something a contradiction. Rerun on the same three planted drafts, it
caught 12 of 12 and showed 14 contradictions, none of them wrong: 11 of the plants (it filed the
twelfth as a tension) and 3 real ones nobody had planted. It kept 10 weaker findings out of the way
and quoted Claude's text as mine 0 times.

**Limits.** One person, three tasks per type and one run each, so a difference of one or two
between C and D is inside the noise: C over D is a lean, not a result. I designed the test, and
the graders were the same model family as the agents being graded. C's tags were inferred by
script, with known mis-tags. B kept some text in my voice that a clean summaries-only wiki might
not have, which makes it realistic rather than clean. The planted contradictions were not reviewed
by me before they were frozen, and one of them could be spotted from the draft alone.

## Connections

**Related:** [The spark has to come from a person](https://latentmirror.com/posts/the-spark-has-to-come-from-a-person/), [The empty knowledge base](https://latentmirror.com/posts/the-empty-knowledge-base/), [The machine and the mirror](https://latentmirror.com/posts/the-machine-and-the-mirror/), [LLM Wiki](https://latentmirror.com/reflections/karpathy-llm-wiki/), [AI Stories Scored Higher. Human Authorship Still Mattered.](https://latentmirror.com/reflections/grove-white-ai-writing-authenticity/)