A generative AI (GenAI) chatbot developed by New York City is under fire after it advised small business owners to break the law.
The “MyCity” chatbot – powered by Microsoft’s Azure AI services – also misstated local policies.
When breaking the news, The Markup quoted a local housing policy expert who said that the bot's information could be incomplete and – at times - "dangerously inaccurate."
Moreover, the bot shared insights on housing policy, worker rights, and rules for entrepreneurs while appearing authoritative – according to the publication.
For instance, it answered questions like: "Do I have to accept tenants on rental assistance?", and: "Are buildings required to accept section 8 vouchers?", with a definitive "no".
In doing so, it inferred that landlords don’t need to accept such tenants.
However, it is illegal for landlords to discriminate on source of income in New York City, with a small exception for small buildings where the landlord or their family lives.
Elsewhere, the bot answered the question: "Can I make my store cashless?", with a "yes". However, since 2020, stores in New York City have been required to accept cash as payment.
Yet, these are just two examples of many, and after reports broke last week, the bot is still available online, giving out false guidance.
While New York City has strengthened its disclaimer by noting that its answers are not legal advice, it continues to embrace the AI system without leveraging sufficient safeguards.
Defending the decision at a press conference on Tuesday, Eric Adams, Mayor of New York City, said:
Anyone that knows technology knows this is how it's done. Only those who are fearful sit-down and say, 'Oh, it is not working the way we want; now we have to run away from it altogether.' I don't live that way.
Julia Stoyanovich, a Computer Science Professor and Director of the Center for Responsible AI at New York University, told AP News that the approach is "reckless and irresponsible."
"They’re rolling out software that is unproven without oversight," continued Stoyanovich. "It’s clear they have no intention of doing what’s responsible."
The bot has been available to the general public since October.
On launch, NYC labeled it a "one-stop-shop" for business owners, generating responses to questions to help steer them through New York City's bureaucratic labyrinth.
