the past few weeks, New generative AI tools, such as OpenAI’s ChatGPT, Microsoft’s Bing chatbot, and Google’s Bard, are all the rage and discussing their potential to transform the way journalists work.
Especially for data and computational journalists, AI tools like ChatGPT have the potential to help with a variety of tasks, such as writing code, scraping PDF files, and translating between programming languages. However, tools like ChatGPT are far from perfect and have been shown to “hallucinate” the data and sprinkle errors throughout the text they generate.
Nicholas Diakopoulos, Associate Professor of Communication Studies and Computer Science at Northwestern University and former Tow Fellow, asked journalists how to avoid these risks and whether ChatGPT could be a useful tool for journalism students and novice programmers. We talked about how to track data from Instructions for using AI.
Diakopoulos recently launched Generative AI in its newsroom project. It explores how journalists can use generative AI responsibly.
As part of this project, news producers are encouraged to submit pitches on how this technology will be used in news production. Find out more about the project here. This conversation has been edited and condensed for clarity.
SG: For journalists and students who are new to programming, how useful do you think ChatGPT is in helping with computational tasks?
ND: As a user myself, I’ve found ChatGPT certainly useful for solving certain kinds of programming challenges. However, we also recognize that it requires a fairly high level of programming proficiency to be able to comprehend it, formulate appropriate queries, and integrate the responses into a practical solution. It can be useful for intermediate coders as long as they know the basics, how to evaluate responses, and how to wrap things up. But if you don’t know how to read the code, it will give you a response, but you won’t know if it’s working as intended.
Programming languages exist for a reason. Because you need to state exactly how the problem should be solved in your code. In natural language, on the other hand, there is a lot of ambiguity. So, obviously ChatGPT is good at trying to disambiguate the question and guess how to give you the code you want, but it’s not always right.
I suspect that journalism students will lose some basic knowledge when using ChatGPT for assignments. When it comes to learning programming, are students better off learning how to code from scratch than relying on her ChatGPT?
One lens through which I look at this issue is AI alternatives and complements. People fear when AI starts talking about replacing someone’s labor. But in reality, most of what we see is AI complementing the work of experts. So you have someone who is already an expert, and the AI marries that person and empowers them to become smarter and more efficient. ChatGPT is a great complement for human coders who have some idea of what they are doing with their code, and I think it can really accelerate your capabilities.
We have started a project in the newsroom called AI where journalists can submit case studies of how they have used ChatGPT within the newsroom. How is that project going?
I reached out to over a dozen people with ideas of varying maturity. From different types of organizations including local news media, national, international and regional publications and start-ups. There are so many people interested in exploring technology and seeing how far it can go for specific use cases. I have contacts with several legal scholars here at the Institute of Information Law, University of Amsterdam, where I am taking a sabbatical. They are looking at copyright and terms of service issues. I know these issues are very relevant and important for practitioners to be aware of.
I’ve also been researching different use cases myself with this technology. I’ve blogged about it and published a pilot project to help people in the community and understand what the features and limitations are. increase. I think things are going well. Hopefully, we’ll see some of these projects mature and start rolling out over the next month.
Now that you’ve seen what journalists are submitting, do you have a better sense of how ChatGPT can help in your newsroom?
There are so many different use cases that people are exploring. I don’t even know if there’s one thing I’m really good at. People are looking for content rewriting, summarization and personalization, news discovery, translation, and engagement journalism. For me, one of the attractions of this project is exploring its scope. Hopefully in a few months these projects will start to mature and get more feedback. I really urge people to evaluate their use cases. For example, how can you be sure it’s working with a level of accuracy and reliability that you can confidently deploy as part of your workflow?
A major concern among computational journalists is that ChatGPT can “hallucinate” the data. For example, when used to extract data from a PDF, everything may work fine on the first page. But when I do that with 2,000 PDFs, suddenly the errors are all over the place. How do you survive that risk?
Accuracy is a core value of journalism. AI and machine learning systems have a statistical element of uncertainty, and it is basically impossible to guarantee 100% accuracy. Therefore, the system should be as accurate as possible. But at the end of the day, even if it is a core value of journalism and something to strive for, whether something he needs to be 100% accurate depends on the information generated from the news. It depends on what kind of claim you want to make with it. AI system.
Therefore, if you need a system that identifies people who are committing fraud based on the analysis of a set of PDF documents, and you plan to publicly prosecute those individuals based on the analysis of those documents, is correct. After years of talking to journalists about stuff like this, journalists probably won’t rely solely on machine learning tools to find evidence of it. They may use it as a starting point. But then they triangulate it with other sources of evidence to increase the level of certainty.
However, there may be other use cases where it doesn’t matter if the error rate is 2% or 5%. Because we see a big trend. Perhaps the trend is so large that even a small error cannot hide it with a 5% error rate. So it’s important to think about your use case and how much error it can tolerate. Then can you figure out how much error this generative AI tool produces? mosquito?
Do you think there will be AI classes or tutorials for journalists in the future on how to use AI responsibly?
We want to avoid a future where people feel they can rely entirely on automation. There may be some hard rules about when you should check the output manually and when you shouldn’t. But I would like to think that many things lie between these two extremes. The Association of Professional Journalists media ethics This is basically all of the case studies and reflections on different types of journalism ethics. It may be interesting to think this way. Perhaps that book needs a chapter on AI, and we should start parsing out what situations are more likely to cause problems and in which situations they are less likely to occur.
Perhaps it wouldn’t be all that different from the way it is today, with core components of journalism such as accuracy and do-no-harm principles. The goal when disclosing information is to balance the public interest value of the information with the harm it may cause to innocent people. Therefore, we need to balance these two elements. Applying such a rubric might make sense when thinking about errors from AI that summarizes something or from generative AI. Likewise, what are the potential harms that could result from this error? Who could be harmed by the information? What harm could the information be harmed?
Yes, journalists make mistakes when working with data too.
However, there are differences. Back to the issue of accountability. There is very clear accountability when humans make mistakes. Can someone explain their process and understand why I missed this thing or made a mistake. It doesn’t mean there is. Tracking human accountability via AI systems is much more complicated.
If a generative AI system makes an error in summarization, we can blame Open AI for creating the AI system. However, when using the system, you also agree to the terms of use and are responsible for the accuracy of the output. So Open AI says it’s your responsibility as a user and they assign you responsibility. They don’t want to take responsibility for their mistakes. And contractually they are obligated to hold you accountable. So now it’s your problem. Are you willing to take responsibility and accountability, for example, like the journalists and news outlets who use the tool?
How do journalists track the use of AI if they have to track errors?
that’s a great question. Tracking prompts is one way to think about it. So, as a user of technology, I have the concept of what is my role in using technology. What parameters did you use to drive the technology? That’s at least a starting point. So if you do something irresponsible at the prompt, you have an example of negligence. For example, if I prompt something to summarize a document, I set the temperature to 0.9 t and a higher temperature means a lot more randomness in the output.
You should be aware of that when using these models. You should know that setting the temperature higher will cause more noise in the output. So if there is an error in that output, you may be held accountable. You may need to set the temperature to zero or much lower to reduce the possibility of randomness in the output. As a user, I think you should take responsibility for how you are prompted, what parameters you choose, and be prepared to explain how to use the technology.
Sarah Grevy Gottfredsen
