Artificial Intelligence Hallucination Raises Alarm After Nearly Triggering Military Action in the USA

Governments worldwide are all pushing heavily to integrate artificial intelligence (AI) into defense and warfare. And why not? It offers a wide range of benefits, including faster decision-making, improved situational awareness, reduced human combat risk, and more. Another key driver is the fierce AI dominance race that nations like the USA and China want to win. But rapid adoption without appropriate safeguards can have devastating consequences, as a recent example of artificial intelligence hallucination shows.

Put simply, hallucination is when an AI model represents false or inaccurate information as a fact with complete confidence. CNN recently reported that an AI-based chatbot used by a Special Operations Command analyst hallucinated and generated a false report that almost led to a USA operation on a Chinese vessel.

The incident occurred this spring during the war with Iran, four sources familiar with the incident told CNN in an exclusive. The analyst used the chatbot to run intelligence reports on the Chinese ship’s manifest. It analyzed that the vessel was carrying components of a nuclear weapons program. The analyst then ran the tool again to summarize the report and send it across.

The US military began preparing to intercept the ship soon after receiving the intelligence report. Military planes were in the air, and armed soldiers were preparing to board a naval ship.

But before proceeding, officials reviewed the report thoroughly. They learned the report was generated using artificial intelligence and that the content inside the Chinese vessel was misidentified. It is a classic example of how artificial intelligence hallucinations can lead to falsified responses.

One source speaking with CNN said the report was completely false. But it also “almost started a war” between the two nations. It was not clear, though, whether the chatbot was commercial or provided by the military.

While commercial AI chatbots may not be meant for military operations and analysis, there are several government-provided tools, too.

A key example of this is the GenAI.mil platform, which the US Department of War introduced in December 2025. At the time, the department had introduced its special version of Gemini for data processing and conversations with artificial intelligence models. It is clear for Impact Level 5 use.

“I am pleased to introduce GenAI.mil, a secure generative [artificial intelligence] platform for every member of the Department of War. With this launch, we are taking a giant step toward mass AI adoption across the department. This tool marks the beginning of a new era, where every member of our workforce can be more efficient and impactful.”

Secretary of War Pete Hegseth wrote in his message to the workforce.

The platform recently introduced a version of Starshield AI’s Grok for Government and OpenAI’s ChatGPT Mil, too.

There’s nothing wrong with using these tools, but you need to understand the uncertainty they carry. For instance, there’s no clear way to understand how an AI solution arrives at a decision or response. It’s all part of the AI black box.

Research is ongoing to address the AI black box. Anthropic’s researchers, for example, found a way to trace a large language model’s (LLM) thought process. Specialized startups like Goodfire AI also focus on building interpretability platforms that make neural networks transparent and steerable.

Engineers are also trying architectural models and techniques like Retrieval-Augmented Generation (RAG) to minimize artificial intelligence hallucinations. RAG improves LLM responses by fetching relevant information from recent and updated external resources before generating a response. Thus, instead of relying solely on their training data, LLMs can use updated knowledge banks for more accurate, better outcomes.

But until there’s a clear path to understanding AI responses, hallucinations will remain a major problem. Therefore, the best practice is to start small and go slow instead of jumping on the bandwagon to enter the fierce race for AI dominance. But governments and businesses will both need to be well prepared to adopt artificial intelligence and implement it correctly.