Mamdani to kill New York City AI chatbot that tells companies to violate laws – The Markup

The Markup, now part of CalMatters, uses investigative reporting, data analysis, and software engineering to challenge technology in the public interest. register to Klaxona newsletter that delivers stories and tools straight to your inbox.

This article is a joint report with Citya nonprofit newsroom serving the people of New York. Sign up for our newsletter scoop.

At a press conference this week on New York City’s $12 billion budget gap, Mayor Zoran Mamdani highlighted the previous administration’s artificial intelligence chatbots as one of the “many things we’re pursuing to save money.”

The chatbot, released by the Eric Adams administration in the fall of 2023, was intended to provide an accessible way for business owners to review the city’s rules and regulations. But as first documented by The Markup and THE CITY, the bot provided answers that, if followed, could lead to illegal actions by the company, including taking a portion of the employee’s tips.

The mayor’s spokeswoman, Dora Pekek, confirmed in a text message that the new administration plans to eliminate chatbots. She said members of the Mamdani transition team saw reports about the bots from The Markup and The City and presented them to the mayor as potential places to save money.

At a press conference, Mamdani blamed Adams for the budget shortfall, saying he had been handed a “poisoned chalice.” He said he would raise taxes on the wealthy and businesses to eliminate the deficit and look at potential savings “under the hood” of the city’s budget.

When asked by reporters what he would cut, he cited chatbots.

“The previous government had an AI chatbot that was functionally unusable,” Mamdani said. “It cost the administration about $500,000. That in itself is not something that can bridge these kinds of gaps, but it does show how the funds have been spent while refusing to account for the real costs of these programs.”

The bot was built using Microsoft’s cloud computing platform and was part of an ambitious overhaul of New York’s digital services called MyCity. The project was intended to streamline access to government, but was criticized for relying on outside contractors.

It wasn’t clear how much it would cost to maintain the chatbot. Just building the infrastructure for the bot reportedly costs close to $600,000, which is close to the amount offered by Mamdani. Pekek said no date has been set yet for removing the bots.

Tests by The Markup and THE CITY in 2024 found that despite the Adams administration’s promises, chatbots confidently provided inaccurate and potentially harmful information to visitors, even on high-stakes topics.

For example, when asked about housing policy, Bott suggested that landlords may discriminate against tenants with Section 8 vouchers. Despite being a resource for business owners, the bot didn’t know about the minimum wage and told users it was okay to refuse to pay them in cash, despite a 2020 city law to the contrary.

After The Markup and THE CITY’s first report was published, readers continued to ask the bot sometimes farcical questions, which the bot was unable to properly answer. The Adams administration defended the bot, saying it would improve over time.

“We’re going to identify what the problem is, fix it, and have the best chatbot system in the world,” Adams said at a press conference. “People will come to see what we’re doing in New York City.”

City administrators quickly added a disclaimer to the bot, advising users not to “use the bot’s responses as legal or professional advice.” We’ve also improved some of the bot’s answers, but it also appears to limit the types of questions the tool can answer.

The bot currently advises visitors to “ask questions only to New York City government” and warns that “answers may produce inaccurate or incomplete content.” Visitors must agree to accept the Bot’s limitations before using the Bot.