Simply put
- Over 40 top AI researchers have proposed to monitor chatbots' internal “chain of thoughts” to capture harmful intent before they become action.
- Privacy experts warn that monitoring these AI thought processes can make it possible to expose sensitive user data and create new risks of surveillance or misuse.
- Researchers and critics alike agree that strict protection and transparency are needed to prevent this safety tool from becoming a privacy threat.
Forty world's top AI researchers have published a paper that argued that companies need to start reading the ideas of AI systems. Instead of their output, the actual step-by-step inference process, internal monologues that occur before ChatGpt or Claude, give the answer.
The proposal, called the Chain of Thinking Monitoring, aims to prevent fraud even before the model has answered, and researchers argue that it can help companies set scores “in training and deployment decisions.”
However, there is a catch to make someone who has nervously entered private questions chatgpt. If a company is interacting with AI with users, if it is able to monitor AI's thoughts in its deployment, it can monitor others too.
When safety is monitored
“The concerns are justified,” said Nick Adams, CEO of commercial hacking startup 0 RCUS. Decryption. “Raw beds often contain verbatim user secrets because they “think” with the same tokens that models ingest. ”
Everything you input into AI goes through that series of thought. Health concerns, financial troubles, confessions – if Cott's monitoring is not properly controlled, it can all be recorded and analyzed.
“History is the skeptic side,” Adams warned. “Telecom metadata after 9/11 and ISP traffic logs after the Communications Act of 1996 were introduced “for security” and later reused for commercial analysis and subpoenas. The same gravity will be drawn to the COT archive, unless the anchorage is encrypted and access is legally constrained. ”
Career nomad CEO Patrice Williams Lind is also cautious about the risks of this approach.
“We've seen this playbook before. Do you remember when social media started with “connecting your friends” and became a surveillance economy? Decryption.
She predicts the future of “consent theatres,” where “companies pretend to respect privacy, but fill in crib surveillance on the 40-page terms.”
“Without the Global Guardrail, COT logs are used for everything from ad targeting to “employee risk profiling” in enterprise tools. Be especially careful with HR technology and productivity AI. ”
The technical reality is particularly concerned about this. LLM is capable of sophisticated multi-step inference only when using COT. The more powerful AI, the more monitoring is required and more invasive.
Furthermore, existing COT monitorability can be extremely vulnerable.
High metrics RL, alternative model architectures, and specific forms of process oversight can all lead to models that obfuscate thoughts.
– Bowen Baker (@bobabowen) July 15th, 2025
While Google's design leader Tej Kalianda is not opposed to the proposition, it emphasizes the importance of transparency so that users can feel comfortable with what AI does.
“The users don't need a full model inside, but she said from the AI chatbot, “This is why I'm looking at this” or “There are things I can't say anymore,” she said. Decryption. “A good design can make the black box feel like a window.”
She added: “Traditional search engines like Google Search allow users to see the source of each result. Click to see the reliability of your site and make your own decisions. That transparency gives users a sense of agency and confidence.
Is there a safe way to do this in the future?
In the name of safety, companies may be able to opt out of providing data for training by users, but these conditions may not necessarily apply to the model's chain of thinking. This is an AI output that is not controlled by the user, and AI models usually replicate the information they provide to make appropriate inferences.
So, is there a solution to increase safety without compromising privacy?
Adams proposed safeguards. “Relaxation: Memory traces with zero-day retention, deterministic hashing of PII before saving, deterministic hashing of PII in a comprehensive analysis.”
But Williams Lind remains skeptical. “We need accountable AI, not performance. That means transparency through design, not monitoring by default.”
For users, this is not an issue for now, but if it's not implemented properly. The same technology that can prevent AI disasters can turn every chatbot conversation into logged, analysed, and potentially monetized data points.
As Adams warned, beware of “a raw cot exposure violation,” “a public benchmark that shows 90% avoidance despite surveillance, or a new EU or California law that classifies COT as protected personal data.”
Researchers are looking for safeguards such as minimizing data, transparency regarding logging, and rapid deletion of non-flag data. However, to implement these, you need to trust the same company that controls monitoring.
But as these systems become more capable, who will see their watchers when they can read our thoughts?
Generally intelligent Newsletter
A weekly AI journey narrated by Gen, a generator AI model.
