Recently, I wrote a piece for the journal on compassion and its uses. I worked as I often do for these pieces: AI co-work. To maintain a writing cadence that lets me post every two weeks, on top of my clinical, teaching, and coaching schedule, I must use AI to supplement my work to some degree. Unlike the work I submit for peer review, this work is lighter and less rigorous. I also spend a lot of time on the writing to get it just right. Weirdly, I spent more time on that piece than I did on this one, which is wholly human-authored. As I was writing that piece, I came back to a draft after a week or so and looked it over and said out loud, “Some of this looks like AI BS”.
But my friend Daniel Reeves had just posted about Pangram, the AI detection software. It piqued my interest, as Danny reported that it was really hard to fool. As Danny is a person I hold in deep esteem, as he is both smart and discerning about these things, I gave it a whirl. It… got it kinda right? While it identified some sections as AI, it also missed others in a weird way. Part of it was that the tool did a binary assessment of chunks of writing, and I was doing highly specific AI co-writing, adjusting the material in free flow to adapt generated passages to the true intent of my work. So I figured I would conduct a thorough coding of the AI work in that draft (let’s be clear, there are many after it, as you will notice in the final copy from a few weeks ago). That is what I put alongside the Pangram ID. Then I did a quick analysis of the coding/ID process for each set.
The document is here; check it out for sources and data.
So there it is, folks, a full transparency analysis. Pangram indicated that 69% of the text was AI-generated. In the end, the actual word provenance was more complex. 16% of this text was my unadulterated wording. Almost 47% of it was touched in some way by computer adjustment. Many of these adjustments were word order, grammar, or rearrangement of structure practices in the vein of advanced grammar or proofreading software. 10.4% of it was total AI generation without much human input. Finally, just under 27% of it was AI-generated text based on my prompts directing it to condense papers or scholarship in a specific way (e.g “include the Sinclair article in [this section] and emphasize the complexity of measuring and defining compassion fatigue as a research concept”). So if you read this as “any AI influence about 84% of the document was AI-retouched, " that's higher than POangram’s estimate. If you read this as “AI but not grammar, spelling, and proofreading,” then the number is more like 10-28% of the text. In either instance, the tool is less precise than I would want and does not seem to account for the way that AI co-writing is emerging in the field, as I see students and colleagues using it.
Really interested in people’s experience of this piece; feel free to reach out and talk personally or respond on socials.
