why should i even, i could be watching that pink flamingoes movie
I absolutely would not want to get in the way of you doing that: In 1989 I missed an all-night session of John Walters movies at the Scala in King’s Cross and I am still full of pointless, futile regret. Don’t make my mistake. Go, with my blessing.
ok i’m back now
Did you enjoy it?
yes
I am so glad to hear that! Although it’s not a surprise, because everyone enjoys Pink Flamingoes.
And, similarly, everyone is quite spectacularly confused about what documentation should look like in a world where obedient LLM servants are churning out text on demand. In particular: which bits should people still do, and what should be left to the machines?
i too am confused about this. can you enlighten me?
I’ll do my best! But this will involve me inflicting some concepts on you.
what concepts will you be inflicting upon me?
It’ll be all about Diataxis, which is a classification of documentation into 4 types. You can see a definition here: https://diataxis.fr/ . It’s not a very new idea – evolving from maybe 2014 onwards. It’s gained a fair bit of popularity already, but I reckon that the advent of LLMs will boost it further. Why? Because, on the LLMs-and-documentation front, it can help us decide what we should leave to LLMs, and what we should still put the effort in on working out for ourselves.
why will knowing about Diataxis help me?
Diataxis helps us answer a crucial question for LLMs: how important is the viewpoint of the human reader?
This matters because LLMs don’t understand people. LLMs are great if the identity of the human reading the documentation doesn’t matter. But if the human viewpoint is relevant, AI output will probably come across as alienating slop.
so what, exactly, is diataxis?
Diataxis splits documentation across two axes: (1) acquiring knowledge vs applying knowledge (2) thinking about things (cognition) vs doing things (action).
This results in the following 4 categories:
- tutorials: acquiring, action: you learn the basics by following lesson steps
- explanations: acquiring, cognition: you gain understanding by reading in-depth context
- how-to-guides: applying, action: you carry out goal-oriented actions by following process-specific steps
- reference: applying, cognition: you access goal-oriented information by reading context-free details
why should i care?
It gets interesting when you start thinking about this question: how does the Diataxis split interact with LLMs, and the related human viewpoint criterion?
ok so how do things line up?
In one line:
(LLMs) = References … How-tos … Tutorials … Expanations = (Humans)
References are a dry list of facts. There is no human viewpoint required; LLMs can assemble this information on demand. Humans may need to provide some sparse signposting so the LLMs don’t burn too many tokens figuring out where to look for the information, but that’s about it.
How-to guides need more human involvement: humans decide what goals are important. LLMs should figure out how any given action towards a goal should be carried out. This leaves a wide zone in the middle – what the actions should be for a given goal. Who should figure that out? The right answer will depend on a number of factors, including: how good the LLMs are, how consistent the codebase is, how many tradeoffs there are between plausible choices.
Tutorials need yet more human involvement: not only do humans still need to pronounce upon the goals (what should people be trying to learn?) but they now need to pronounce upon the actions (what actions will teach people what they need to learn?). The role of the LLMs shrinks, but they are still useful to check the correctness of the actions; they can also be used to check for potential gaps in understanding (but a human is needed to pronounce upon whether these LLM-identified gaps are meaningful).
Explanations should be predominantly human-curated. The whole purpose of an explanation is to provide deep context: knowledge that can’t be gleaned from inspecting the codebase, because it depends on what human beings think is important – a good example of this deciding to what extent it’s useful to provide historical context for the current state of the system. The role of LLMs should be confined to fact-checking only. Don’t get an LLM to write your explanations for you.
so is diataxis the only way i should think about docs?
No. Documentation is ultimately about using permanent meda to convey information. The degrees of freedom involved are unfathomably complex. We should be humble in the face of this complexity: we shouldn’t expect Diataxis to have all the answers.
oh fgs i am an engineer so for the love of god give me a concrete example that i can respect rather than waffling like a stoned maths major who just found their first ever semiotics textbook in a second hand bookshop
My past self feels for you. Here’s a specific example of how Diataxis falls short: you have to work to avoid blurring and mixing between “how-to guides” and “explanations”.
um ok explain
Whether we understand some area of knowledge isn’t a binary yes-or-no thing, right? We might get the whole lot, but it’s more likely we’re patchy – we get almost all in some areas, very little in others. But a Diataxis “how-to guide” assumes that the reader already understands all the relevant concepts involved in carrying it out.
So there’s a gap there. Practically speaking, you often end up with bits of explanations mixed into the “how-to guides” in an attempt to make them accessible to as broad an audience as possible.
Makes sense, right?
so how does diataxis deal with that?
It doesn’t! The Diataxis site instead spends a fair bit of time dealing with two other distinctions that I personally find far less tricky to deal with: tutorials vs how-to guides and reference vs explanation (see https://diataxis.fr/theory/).
That the Diataxis site is preoccupied with those two other distinctions makes sense in terms of the Diataxis set up: each of those two pairs differs in only one of the two Diataxis split points, so you would expect some confusion at the boundary.
But “how-to guides” and “explanations” are diametric opposites in the Diataxis taxonomy – they aren’t supposed to get confused, at least according to the Diataxis split’s view of reality!
so is diataxis wrong then?
No, it’s just not perfect. But so few things in this world are perfect. Your child’s first smile, sure. The night sky in the Sahara, perhaps. The glint in Divine’s eye in that scene from Pink Flamingoes, absolutely. But most of the world falls short of such exacting standards.
For documenting information technology tools, Diataxis still empirically works very well! It’s interesting that there is some signal that its theoretical underpinning has some deficiencies, but that’s no reason not to go with Diataxis – it seems to be a productive way of channelling documentation effort.
does something need fixing here?
Maybe? How big the problem is depends on how complex your systems are. My intuition is that if things scale up, you’d need to pick some other conceptual framework to gain useful insights. Diataxis has a certain realm of validity, and if you want to get maximal milage out of a model, you need to know where its limits are.
i should have just watched another movie
I’ve put a whole load of stuff across here. Too much maybe. It might help you to chill out and read a book. Might I recommend Cities of the Red Night? It’s a compelling tale of high-seas adventure and civic pride, filtered through a distinctive modern sensibility. Easy reading.
thank you
You’re so welcome x
i just read that book
are we still friends
no
