why should i even, i could be watching pink flamingoes
I absolutely would not want to get in the way of you doing that: I missed an all-night session of John Walters over 30 years ago and am still full of regret. Go, with my blessing.
ok i’m back now
Did you enjoy it?
yes
That’s good! Although it’s not a surprise, because everyone enjoys Pink Flamingoes.
And, similarly, everyone is quite spectacularly confused about what documentation should look like in a world where obedient LLM servants are churning out text on demand. In particular: which bits should people still do, and what should be left to the machines?
i too am confused about this. can you enlighten me?
I’ll do my best! But this will involve Concepts.
what concepts will you be inflicting upon me?
It’ll be all about Diataxis, ex a way of classifying documentation into 4 types: see here for a definition of Diataxis if you haven’t come across it already: https://diataxis.fr/ . It’s not a very new idea – evolving from maybe 2014 onwards – but it’s already gained many adherents, and I think that it’s about to get something of a boost. This is because we all realise that having LLMs to help with serving up documentation means that we can ditch a lot of the existing static content. But it’s hard to figure out what to ditch – what still needs human involvement? And it turns out that the Diataxis split can help us with that.
how does knowing about Diataxis help?
The Diataxis split helps with a crucial question relevant to where to use LLMs: to what degree some given documentation depends upon being written from a particular human viewpoint. The less you need to know about the human reading the documentation, the greater the role LLMs can play. LLMs are magical at churning through text, but they don’t know who you are.
so what, exactly, is diataxis?
Diataxis splits documentation across two axes: (1) acquiring knowledge vs applying knowledge (2) thinking about things (cognition) vs doing things (action).
This results in the following 4 categories:
- tutorials: acquiring, action: you follow lesson steps to gain introductory knowledge
- explanations: acquiring, cognition: you read to gain in-depth context
- how-to-guides: applying, action: you follow process-specific steps to achieve some goal
- reference: applying, cognition: you read to access context-free details
whatever. so what’s the big question?
It is: how does the Diataxis split interact with LLMs, and the related human viewpoint criterion?
get on with it, i’m getting bored
Let’s start with an easy win: for one of the four categories of Diataxis, there is no human viewpoint, and so no need for any persistent documentation. This is the reference category – a reference simply presents all the information, without any assumption about what you will want to do with it. In a world where LLMs assist in serving up documentation, references effectively cease to exist in any permanent form. You will simply ask an LLM pointing to your codebase and config to serve you the information up dynamically instead. Human input will be minimal – perhaps a standard prompt that enables the LLM to efficiently discover the information.
Tutorials and How-to guides are next in terms of how much LLMs can do the heavy lifting – these provide sets of actions to follow for people who either want to learn something from scratch (tutorials), or to achieve some specific end (how-to guides). Right ar the start, the goals and their consequent series of actions need to he selected by humans. But whether each individual action is correctly described can be validated and kept up-to-date by LLMs – decent ones can probably write much of the actions for you too, particularly if you have a well-structured code base and some decent explanatory material where required.
Ultimately it’s only explanations that remain predominantly human-curated, although one can (and should) use LLMs to do fact-checking on them. This is because the whole purpose of an explanation is to provide deep context: knowledge that can’t be gleaned from inspecting the codebase, because it depends on what human beings think is important.
so is diataxis the only way i should think about docs?
No. Documentation is ultimately about using writing to convey information. The conceptual world of writing prose is … large … no one model will capture all of that complexity. We shouldn’t expect Diataxis to have all the answers.
oh blah blah blah i am an engineer, give me a concrete example rather than waffling like a gcse english teacher who went to the pub at lunchtime
Here’s a specific example of how Diataxis falls short: you have to work to avoid blurring and mixing between “how-to guides” and “explanations”.
Why does this point to a problem with Diataxis? First of all, let’s establish that it’s a genuine issue.
Understanding some area of knowledge isn’t a binary yes-or-no thing. Some of us get the whole lot, some of us have a more patchy understanding. But any “how-to guide” in the Diataxis franework should assume that the reader already understands all the relevant concepts involved in carrying it out.
Bridging this gap – between the binary knowledge assumption of Diataxis-style “how-to guides” and the more nuanced levels of understanding present in the real world – gives rise to cross-talk between “how-to guides” and “explanations”. Practically speaking, you often end up with bits of explanations mixed into the “how-to guides” in an attempt to make them accessible to as broad an audience as possible.
Makes sense, right?
so how does diataxis deal with that?
It doesn’t! The Diataxis site instead spends a fair bit of time dealing with two other distinctions that I personally find far less tricky to deal with: tutorials vs how-to guides and reference vs explanation (see https://diataxis.fr/theory/).
That the Diataxis site is preoccupied with those two other distinctions makes sense in terms of the Diataxis set up, Each of those two pairs differs in only one of the two Diataxis split points, so you would expect some confusion at the boundary. By contrast, “how-to guides” and “explanations” are diametric opposites in the Diataxis taxonomy – they aren’t supposed to get confused, at least according to the Diataxis split!
so is diataxis wrong then?
No, all this just means that Diataxis isn’t perfect. And very few things in this world are perfect. Your child’s first smile, sure. The night sky in the Sahara, perhaps. The glint in Divine’s eye in that scene from Pink Flamingoes, absolutely. But most of the world falls short of such exacting standards. For documenting information technology tools, Diataxis still empirically works very well! It’s interesting that there is some signal that its theoretical underpinning has some deficiencies, but that’s no reason not to go with Diataxis – it seems to be a productive way of channelling documentation effort.
The cross-chat between “how-to guides” and “explanations” points to some kind of problem in the way Diataxis attempts to cleave reality: a misalignment between the binary “you know it or you don’t” assumption of “how-to guides” and the actual nature of undersanding for complex systems that “explanations” attempt to deal with. This problem will become worse as the system being described becomes more complex, covering a wider variety of more richly interrelated conceptual setups. There’s maybe something missing in the Diataxis system about how that can be managed.