The HaLLMark Effect
Illuminating the intersection of human/AI co-writing
The rise of Large Language Models (LLMs) like GPT-4 has sparked both excitement and concern in the realm of creative writing. These powerful tools can assist in generating content, offering new ideas, and overcoming writer’s block. However, there are valid concerns about the authenticity and value of AI-assisted writing. And besides, if the LLM wrote your paper, do you even feel a sense of ownership in it — and do you deserve to?
Enter HaLLMark, our new tool designed to address these issues by providing transparent and interactive visualizations of the AI’s contributions in the writing process. We presented a paper describing this tool and its evaluation at the recent ACM CHI 2024 conference in May.
The human/AI co-writing dilemma largely lies in balancing the efficiency and creative boost offered by AI with the need for genuine human authorship and ownership. This is where HaLLMark steps in by providing a transparent visualization-based interface that helps writers track their interactions with AI, ensuring they remain in control. The tool integrates an AI interaction history with the writing process, visualizing how much of the text is influenced or generated by the AI. This transparency allows writers to maintain a sense of control and ownership over their work; a crucial aspect of human-centered AI tools.
The most straightforward visualization mechanism in HaLLMark is text highlighting: human-generated text is highlighted in green, while AI-generated text appears in orange. This immediate visual feedback helps writers see at a glance how much of their work is original versus AI-assisted.
The tool also provides a visual timeline of AI prompts and responses, showing the evolution of the text. Writers can hover over any point in the timeline to see the specific AI prompt and the corresponding generated text. This provenance feature not only aids in transparency but also helps in understanding the AI’s impact on the writing process.
Beyond this, HaLLMark offers summary statistics, such as the percentage of text written by the AI and the number of AI prompts used. These metrics give writers a clear overview of their writing process, reinforcing their control and agency.
In a study involving 13 creative writers, HaLLMark demonstrated significant benefits over a “vanilla” ChatGPT session. Writers reported feeling more in control and better able to evaluate the AI’s assistance. The tool helped them to limit AI influence when necessary, ensuring that the final piece remained true to their creative vision. Moreover, HaLLMark made it easier for writers to communicate the extent of AI involvement to publishers and readers, promoting transparency and trust.
A real-world use of a tool similar to HaLLMark may be to integrate it with word processors such as Microsoft Word or Google Docs. Additionally, we are considering expanding the tool’s features to include more nuanced tracking and reporting options to further enhance its utility.
Citation
- Md Naimul Hoque, Tasfia Mashiat, Bhavya Ghai, Cecilia Shelton, Fanny Chevalier, Kari Kraus, and Niklas Elmqvist. (2024). The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1045:1–1045:15, https://doi.org/10.1145/3613904.3641895 (arXiv)
