Can AI Replace Long-Form Research Methods?

Historically, I haven’t been a huge fan of LLMs. AI and machine learning are fascinating topics, sure, but I have some issues with them.

The way the companies market them is too aggressive and just makes me more skeptical. I don’t think the quality of their output is very good. I have some concerns about their impact on the environment, and I’m not terribly keen on the whole argument that certain uses/training methods are okay with the law “as long as it’s transformative” thing.

It’s supposed to make your life easier or boost productivity. But the whole thing is just weird to me, especially as the narrative around it keeps changing. So, in the spirit of keeping an open mind and partially to see what all the fuss is about, I was challenged to do a John Henry-style “battle” with an LLM concerning my research methods.

What happened honestly kind of shocked me.

I’ve been chasing the end of the long tail for a long time

A little bit of background on my research methods

I’ve been researching and writing long-form content for a long time. I worked in TV and radio in the early 2000s, pivoted to newspapers, and started freelancing in 2015. I have written for several outlets since, mostly longer pieces that require interviews, extensive research, and actively participating in or experimenting with the technology or music I’m writing about. I have a degree in journalism and electronic media. My minor was focused on statistics, analytics, and audience research methodology.

I even tackled the subject of LLMs a few years ago, when they were far less established. It was interesting, but I never found them very useful and have not incorporated them much into my work.

With the way things have been going lately, I began thinking about whether these chatbots and LLMs can do half of what they say they can. Claude seemed to be the most highly recommended to me, so I figured, “Why not? I’ll give it a shot and see how it goes.”

For online searches, I use a mix of different methods. Here are a few (this is not an exhaustive list):

• The Slop Evader extension
• Kagi search
• Internet Archive and Wayback Machine
• Wikipedia
• Metafilter
• Advanced searches on various search engines
• Older versions of Encarta and other CD-based encyclopedias
• Physical encyclopedias, atlases, and books when available
• Magazine or book excerpts, if available
• Interview transcripts, if applicable

I chose Claude for this experiment. I hate ChatGPT for reasons I won’t get into here and Gemini is always up my butt about wanting to do everything for me. Claude is a bit more grounded. So I gave it an “absolute” personality to guide it as much as possible as well as setting up a skill for it to use.

Hallucinations are still a problem

LLMs often haven’t felt useful to me and I’ve been very skeptical of them

With the exception of experimenting with the models a bit over the past few years, I don’t typically incorporate LLMs into my workflow. Most of the time, I’ve found alternative or traditional methods of ideation, outlining, and research to be better.

In the interest of full disclosure: yes, I have used LLMs for some work-related tasks as required by a particular company. It’s usually limited to outlining and brainstorming. I don’t like the way they write very much, although they’re good at clinical ad copy, I guess. I am not a fan of AI art and don’t even get me started on the CEO who said musicians don’t like to make music (seriously? Has he ever met a musician?).

I also don’t trust a product that promises to be a fountain of knowledge yet comes with a disclaimer telling me it “can make mistakes” and that I should double-check its work.

Plus, Claude straight up told me it would probably hallucinate during this experiment, which cracked me up.

Hey, at least it’s honest.

Long-form research with Claude

A John Henry-style showdown between man and machine with surprising results

Over the years, I’ve developed a mix of research methods that combine Gonzo journalism, traditional research, library research, primary sources, and hands-on experience. I read articles, journals, and books. The research, for me anyway, is half the fun.

For example, when I wrote about Synthesizer guitars, I spent considerable time actually playing different guitars. That was on top of combing through old guitar player magazines, reading the manuals, and learning what I could online about the subject.

I read interviews with Ryo Kawasaki and Allan Holdsworth. I talked to a professional musician I know who used them back in the 80s. And I took an excessive amount of notes before ever writing a word. Whenever I wrote about novelty songs, I always read liner notes and reached out to the songwriters whenever possible. Nardwuar rejected my interview request, but you get the point: I try to do what I can to be as thorough and accurate as possible.

That’s what I mean when I say “my research methods.” I also used to be quite good at “extreme Googling,” but have largely switched to Kagi for my primary search engine.

It’ll be interesting to see if Claude can hold a candle to these. I’ve always been a tech and music journalist, so those will be the focus areas for this experiment. I devised a research “skill” for Claude and added it to the skill repository. I named it Ziggy, after the computer in Quantum Leap:

Then I broke the challenge down into two research topics: one I know a lot about and one I know very little about at all. I’ll let you guess which one is which.

Comparing long-form research methods with Claude provided unexpected results

Claude helped fill in some gaps for some obscure music-related research

For the first experiment, I wanted to look into an old tour I attended, but I simply couldn’t remember much about it. All I knew was that it featured Lyle Lovett and John Hiatt.

They, along with a few other musicians, would go on stage and trade songs, sometimes accompanying each other. I wanted to 8learn more about the tour and, hopefully, find the show’s set list.

My research involved a lot of advanced searching and following up on leads. In the interest of full disclosure, I performed this research process before asking Claude.

I knew Lyle Lovett and John Hiatt were involved, so the first thing I did was go to his official website and check out previous tours. Since that only turned up current tour information, I had a few options: use a well-known resource like setlist.fm to search, use the Wayback Machine, or add some variable to my search string.

A quick search using the phrase “John Hiatt & Lyle Lovett” brings up a Forbes interview and some advertisements. Yuck. I had more luck at The Internet Archive, where searching for Lyle Lovett + John Hiatt netted a show from 2010 where they performed together. Not quite what I was looking for, but it’s a good start. An advanced search with a date range provided similar results. Eventually, I found some interviews/performances featuring the duo, but not the information I wanted.

On setlist.fm, we had a few more options and could narrow our results by tour. Great. I chose the most likely one, the “songwriters” tour. It yielded two dates, but only partial set lists. So I tried John Hiatt.

Apparently, they were on tour together earlier this year. Still not exactly what I was looking for, but a more thorough search of setlist.fm yielded a few facts:

Lyle and John have played around together often since 1990, at least a few times a year since the 2000s started. Now we were getting somewhere. There were two other songwriters who played with them on the tour I’m thinking of: Guy Clark was one of them. I couldn’t remember who the fourth musician was.

I tried another search, this time on a concert database, and hit pay dirt: the show I was thinking of was from 2004 and featured Lyle and John playing a show with Guy Clark and Joe Ely. I couldn’t find anything about it in any of the guitar/music magazine databases, but with the date, I could narrow things down even further.

From there, I found an interview with Lyle where he talked about the four of them doing “half a dozen shows” in the early 2000s and more set lists on a different concert database. Apparently, they played together like today for a few dates at a time over the course of several years. It turns out the specific show I attended was most likely in 2004 at the Telluride Bluegrass Festival. I was attending the Telluride Bluegrass Festival that year and remember it (mostly) and the research supports my memory.

Armed with this information, I found set lists and even a copy of the show I attended via the Grateful Dead tape-trading community of all places. Not bad for about an hour’s worth of old-fashioned research.

Claude’s research went a lot better than I expected. Here’s the prompt I used:

Claude reported information about the show (and that it was an XM radio broadcast originally, which is confirmed by the taper who made the show available to the community), which turned out to be mostly accurate.

Claude’s findings helped confirm my memory of the show: it was, in fact, a 2004 show at The Telluride Bluegrass Festival. It even provided a link to the correct site showing past festival lineups. The next few links (or “the deep dive”) turned up a few interviews that never came up during my research. That was an unexpected, yet pleasant surprise. It turns out the four musicians continued playing for a few years together and sometimes billed themselves as “Lyle and Friends.”

Again, that was news to me. I liked the way it succinctly provided a hub with links that I could vet. It also followed my instructions about not doing the work for me. Using this as a jumping-off point could work well for getting started on a research project. I worked with Claude to dig deeper into the format, which sent me to a buried NPR podcast from 2007 and more resources that didn’t turn up in my own searches.

It was all very surface level stuff that I would have found over the course of my normal research. While it’s handy to have everything up front like that, it’s more difficult to find something unexpected or unique. I would say it’s not bad for getting started, but I absolutely wouldn’t use it to replace long-form research like this.

The show still holds up, by the way.

Short-form research with Claude

A quick shot of research working with Claude

To wrap up this experiment, I tried out a shorter research question. I selected one based on a news story that I half-remembered from a few years ago. I once read about a man who tried to cross the ocean in a bubble. So I described the story to Claude and asked:

Who was he, how did he do it, and what are some specific details about the vehicle and the journey? Please provide at least three sources for your research. Was it a real thing or inspired by something? Perhaps Claude could fill in some gaps. A quick Google search using the slop evader extension turned up a name: Reza Baluchi, and a vague recollection of a story I read on Vice a few years ago. I read his Wikipedia entry to prep, but I remember he basically tried to cross the ocean in a giant hamster wheel/plastic bubble thing.

Claude hit me with a detailed summary right away, so I started vetting the links. The response confirmed the story and linked to the original Vice article as a source. I found that right away as well, so the start of the process was evenly matched.

Things got even more detailed from there. We got a timeline of events:

And detailed information about his life:

As for the event, it turned out, I remembered something like that happening, but it was a guy in a bubble on the surface of the ocean. My imagination turned him into a nautical explorer at the bottom of the Pacific.

The model output on the short-form topic were promising. My impression is that it largely felt like Claude acted as a junior research assistant and provided some key info with sources that I could then explore myself. It wasn’t trying to do my work for me, and I found it shockingly useful for the start of a research project or generating leads about where to go.

I think it’s a nice touch that Claude will actively engage you in conversation about the topic and prompt you about where to take the research next. That was by design in my instructions, and it handled it well. Which brings me to my next point: Claude helped lead me to these conclusions based on refined model output. It played to my preferences and instructions more than anything else. It isn’t a magic box that did a bunch of work for me. It merely helped support my personal findings in a useful way.

Make sure you’re double-checking Claude’s model output and checking it against multiple, verifiable sources. Don’t merely rely on it to spoon-feed you answers.

By using Claude in conjunction with my typical methods, I managed to get a fairly decent start to researching the topic, so I would rate the short form research a little higher performance-wise than the long form one from before. I will, however, begrudgingly admit that perhaps I was wrong about these models over the past few years.

---

I’ll begrudgingly admit that this could potentially be the future of research

AI, LLMs, and the entire conversation surrounding the topic are truly bizarre. Looking at them as a tool, I will admit to seeing some value with certain things (brainstorming, collecting thoughts, outlining). For its part, Claude did a decent job starting the information-gathering process, but I will always conduct my own research, especially when it comes to interviews and gonzo journalism.

That said, I think its research potential could definitely improve over time, and it genuinely feels like there’s some merit to working better together instead of choosing one method over the other. I’m not saying I’ll do that, I’m just acknowledging its potential. But who knows? I’ll continue to experiment with integrating it into my research process and see where it goes. Claude’s model outputs are much more accurate than I expected.

I still have some concerns about generative AI technology and the conversation surrounding it. I absolutely wouldn’t recommend using it to do everything or outsourcing your thinking to it. In my mind, that just defeats the purpose of the tool. But as a legitimate assistant? There’s certainly potential there.