OpenAI is under intense scrutiny following a deal to provide its technology to the Department of Defense for use within classified military networks. The agreement has triggered resignations and significant debate regarding the company's commitment to safety principles concerning mass surveillance and autonomous weapons.

Caitlin Kalinowski, lead of OpenAI's robotics team, resigned citing concerns that the deal was "rushed without guardrails defined." Her departure highlights a core dispute: whether OpenAI's contract with the Pentagon adequately prevents mass surveillance and the deployment of autonomous weapons.

The deal was announced shortly after Anthropic declined to remove restrictions on government use of fully autonomous weapons and mass surveillance from its own contract. The Trump administration subsequently declared Anthropic a national security risk, banning federal use. OpenAI then announced its own agreement to deploy models in U.S. government classified networks.

OpenAI CEO Sam Altman stated the company's core safety principles-prohibiting domestic mass surveillance and ensuring human responsibility for the use of force-are shared by the Department of Defense and reflected in their agreement. However, publicly released portions of the contract suggest terms that are less restrictive than Anthropic's.

Analysts express concern that the contract's provisions on surveillance leave room for mass data processing and analysis, despite appearing to limit such activities. "They are taking very technical legal words of art such as surveillance... which ordinary people understand in one way and national security lawyers understand in other ways to muddy the waters," noted Brad Carson, president of Americans for Responsible Innovation.

Under national security law, "surveillance" has a narrow technical meaning, primarily focused on intercepting communications of U.S. entities. However, mass data collection from databases and incidental capture of American communications while targeting foreigners fall outside this definition. The "lawful" monitoring of non-Americans could thus yield vast amounts of data on U.S. citizens, made easily accessible by AI systems.

This situation unfolds as the government has been actively purchasing and compiling extensive consumer data, including social media and geolocation information. The capabilities of large language models to analyze this data at scale raise further questions about the definition of "lawful use" by the government.

Following initial announcements, Altman revised contract language, stating, "Our tools will not be used to conduct domestic surveillance of U.S. persons, including through the procurement or use of commercially acquired personal or identifiable information."

Charlie Bullock, a senior research fellow at the Institute for Law and AI, views the revised language as an improvement but acknowledges trust in OpenAI and Altman is paramount. Jessica Tillipman, a law professor at George Washington University, remains skeptical, noting that private parties in government contracts often have limited recourse to withdraw.

OpenAI plans to use its "safety stack" of filters and controls to monitor usage. However, experts like Sarah Shoker, a senior research scholar at the Berkeley Risk and Security Lab, argue that technical controls alone may not be sufficient to prevent misuse, especially given the discovery of "jailbreaks"-methods to bypass safety controls-in various AI systems.

Internal dissent is also significant, with employees expressing concerns. Altman himself admitted the deal "shouldn't have rushed" and appeared "opportunistic and sloppy." Over 100 OpenAI employees have publicly called for refusing domestic mass surveillance and fully autonomous weapons systems.

Tim Marple, a former OpenAI employee, stated that few at the company support facilitating mass surveillance or autonomous weaponry, suggesting a potential "catastrophic loss of labor" if the company proceeds, as engineers could easily move to competitors like Anthropic.

OpenAI's stance on military sales has evolved, initially prohibiting them until January 2024. Despite this, reports indicate the Pentagon was testing OpenAI models through Microsoft as early as 2023. Both OpenAI and Anthropic recently participated in a competition involving AI-coordinated drone swarming technology.

Meanwhile, competition between OpenAI and Anthropic is intensifying, with Anthropic projected to potentially overtake OpenAI in annual revenue by mid-2026.