All That Glitters Is Not GPT
Transcending the LLM gold rush
In 1896, when gold was discovered in the Klondike River, it triggered a massive migration that would become known as the Yukon Gold Rush. Nearly 100,000 prospectors flooded into the region, armed with mere farm tools and wild dreams of striking it rich. Today, we’re witnessing a similar phenomenon in human-computer interaction (HCI), sparked not by precious metals, but by the advent of Large Language Models (LLMs).
The release of GPT-3 in 2020, followed by GPT-3.5 in spring 2022, marked a watershed moment in AI access. For the first time, non-AI researchers and developers had easy API access to capable language models that could understand and generate human-like text. Just as the discovery of gold transformed previously quiet territories into bustling frontiers, these LLMs in many ways turned the HCI research field into a modern-day gold rush. Speaking as one of the IEEE VIS 2024 papers chairs, it is remarkable how quickly LLMs have become a staple of the conference, and recent ACM SIGCHI conference attendees will likely not disagree.
The parallels between historical gold rushes and the current LLM research boom are striking. During the California and Yukon gold rushes, people from all walks of life abandoned their previous pursuits to seek fortune in the goldfields. Similarly, we’re seeing researchers from diverse academic backgrounds pivoting to LLM research, often with limited experience in human-AI interaction.
Like the prospectors of old who arrived with basic pans and pickaxes, many newcomers to LLM research employ rudimentary methods — often just a chat window interface layered on top of GPT. These “thin veneer” papers, which I have talked about before, mirror the gold panning of the 1890s, where early prospectors sought easy pickings before the need for more sophisticated mining techniques became apparent.
The gold rush mentality is evident in other ways too. Just as historical gold rushes created a temporary suspension of the established order — a “wild west” period —academic rigor and methodological standards sometimes seem to take a back seat to the rush for quick publications. Many HCI + LLM researchers are focused on short-term gains rather than building sustainable research programs or developing deeper insights into human-AI interaction. There’s also a parallel in how newcomers relate to established researchers in the field. Akin to how gold rush prospectors often disregarded indigenous knowledge and existing communities, new LLM researchers often overlook decades of prior work in HCI and human-AI interaction.
Before you accuse me of hypocrisy, note that I in no way claim to be an impartial observer. I, myself, participated in this LLM gold rush, and I am just as guilty of some of these transgressions.
Fortunately, nature is healing. Similar to how the historical gold rushes eventually stabilized into organized mining operations, we’re beginning to see the HCI + LLM research field mature. The initial flood of papers featuring basic chat interfaces is giving way to more sophisticated research exploring the deeper implications of human-LLM interaction. Projects like DirectGPT and Smallville’s generative agents demonstrate how HCI research is moving beyond simple implementations to create meaningful advances in how humans and AI systems can work together.
As we move forward, we should remember that HCI is far more than just creating fancy interactive forms to fill in empty fields in a natural language prompt. The field has always been about understanding and improving the complex relationship between humans and technology. LLMs are powerful tools, but they’re just that — tools. The real value lies in thoughtfully integrating AI models into systems that enhance human capabilities and experiences.
The gold rush mentality served its purpose: it brought attention and energy to an important technological advancement. Many of us, myself included, have been caught up in the excitement of these new capabilities. But now it’s time to transition from prospecting to sustainable development. LLMs are here to stay, and our understanding of their proper use (and limitations) continues to grow. The future of HCI research in this space requires careful, methodical work that builds meaningful value on top of these powerful new capabilities.
As we look back on this period, we’ll likely see it as a necessary, if somewhat chaotic, phase in the evolution of human-AI interaction. The challenge now is to channel the enthusiasm and innovation of the gold rush era into more sustainable and substantive research directions. The real gold, after all, isn’t in simply exploiting LLMs for quick gains, but in fundamentally improving how humans and AI systems work together.
