Following California Governor Gavin Newsom vetoing a landmark AI safety bill aimed at regulating the development and deployment of advanced AI systems, we look at the reasons why it was blocked and the implications of doing so.
What Bill?
US Senate ‘Bill 1047’ relates to regulating AI systems with the focus specifically around frontier AI models (highly advanced/cutting edge and large) with the potential for large-scale impact.
California
The fact that California is home to major AI companies like OpenAI (which also partners with Microsoft), and its governor vetoed it, means there are implications for the future of AI governance and industry practices worldwide. For example, Gavin Newsom said in his statement about the bill “California is home to 32 of the world’s 50 leading Al companies, pioneers in one of the most significant technological advances in modern history”.
The Key Points
The key points of the bill were:
– Risk mitigation for frontier AI models. The bill targeted large AI systems, particularly those that required significant computational power to develop (at least 10^26 FLOPS). It required companies developing such systems to implement safeguards to prevent catastrophic harm, including the misuse of AI for creating weapons of mass destruction, committing serious crimes like murder, or launching cyberattacks that could cause significant damage (e.g. over $500 million).
– A “kill switch requirement”. Under the bill, developers would have been required to implement a “kill switch” mechanism to immediately halt the operations of AI models if they posed a threat, during both training and usage.
– Cybersecurity measures. Companies were required to have strict cybersecurity protocols in place to prevent the unauthorised use or modification of these powerful AI systems.
– Oversight and reporting. The bill proposed the creation of a “Board of Frontier Models”, a new state entity, to oversee the compliance of these companies with the safety measures. Regular audits and detailed reports on safety protocols were part of the requirements.
– Whistleblower protections. The bill also included protections for employees who reported non-compliance within their organisations.
Opposition From Big Tech Companies
Major tech companies, including OpenAI, Google, and Meta, strongly opposed the bill, arguing that the regulations could significantly slow down innovation and hinder the deployment of beneficial AI technologies. With these companies being invested heavily in the development of AI, viewing it as a key future revenue source, it’s perhaps not surprising that they saw the bill as a threat to that potential. Also, there were concerns within the tech community that open-source AI models, which often rely on collaborative, decentralised development, could face legal liabilities under the bill’s stringent requirements. This risk, they argued, could discourage further development of open-source AI, which has been an important driver of innovation in the field. The tech giants feared that the bill’s overly strict regulations could stifle growth and limit the industry’s ability to remain competitive globally.
Why Was The Bill Vetoed By Newsom?
In an official statement, the California state governor gave the following main reasons for blocking the bill:
– An overly narrow focus. Newsom argued that the bill only targeted large, expensive AI models based on their computational scale and costs, which could give a false sense of security. He pointed out that smaller, specialised models could pose similar risks but were not covered by the bill.
– A lack of adaptability. The governor emphasised that AI is evolving rapidly, and the bill’s framework was too rigid, not allowing flexibility to adapt to technological advancements. He stressed that regulation needs to be able to keep pace with innovation.
– It ignored deployment context. Newsom criticised SB 1047 for failing to consider where and how AI models are used, whether in high-risk environments or for critical decision-making, arguing that this oversight made the regulation less effective.
– The potential for stifling of innovation. He also expressed concern that the bill could curtail innovation by applying stringent standards to even basic AI systems, which may inhibit the development of AI technologies that benefit the public.
– A lack of empirical evidence. Newsom insisted that any AI regulation must be based on empirical evidence and analysis of AI systems’ actual risks and capabilities. He argued that SB 1047 lacked this necessary foundation.
– Preference for broader collaboration. Instead of a California-only approach, Newsom said he favoured working with federal partners, experts, and institutions to craft a balanced and informed AI regulatory framework.
The Response
Although Newsom’s blockage of the bill may have pleased the big AI companies, not everyone was happy about it. For example, California state Senator Scott Wiener, who represents the 11th district, encompassing San Francisco and parts of San Mateo County, has strongly criticised Governor Newsom’s decision to veto Senate Bill 1047. In a press release, Mr Wiener expressed deep concern about the implications for public safety and AI regulation, arguing that that the bill was designed to introduce commonsense safeguards to protect the public from significant risks posed by advanced AI systems, such as cyberattacks, the creation of biological or chemical weapons, and other harmful applications.
He emphasised that while AI labs have made commitments to monitor and mitigate these risks, voluntary actions are not enforceable, making binding regulation crucial. Senator Wiener said, “This veto leaves us with the troubling reality that companies aiming to create an extremely powerful technology face no binding restrictions from U.S. policymakers,” highlighting the lack of meaningful federal regulation as a critical issue.
Wiener also dismissed the claim that SB 1047 was not based on empirical evidence, calling it “patently absurd,” given that the bill was crafted with input from leading AI experts. Mr Wiener has made it clear that he views the veto as a missed opportunity for California to lead on innovative tech regulation, similar to past actions on data privacy and net neutrality, saying, “We are all less safe as a result.”
However, despite the setback, Wiener expressed hope that the debate has advanced the issue of AI safety globally and vowed to continue working towards effective AI regulation.
Ever-Present AI?
Newsom’s vetoing of the bill comes at the same time as Microsoft’s head of AI, Mustafa Suleyman, saying that he believes that (AI) assistants with a “really good long-term memory” are just a year away in development. Suleyman’s comments refer to “ever present, persistent, very capable co-pilot companions in your everyday life”, which aligns with the idea of the view of many that to make AI truly useful / to leverage the full benefits of AI, integration is necessary. For example, an AI assistant can only organise your schedule if it has full access to your diary and remembers past interactions.
This concept of deeply integrated AI assistants actually ties directly into the debate around Senate Bill 1047, which Governor Gavin Newsom recently vetoed. The bill sought to regulate advanced AI systems, ensuring safety protocols for powerful models. As ever-present AI systems become more common, the absence of legislation like SB 1047 leaves critical questions about how these systems will be governed. Newsom’s veto reflects ongoing concerns about stifling innovation, yet it also leaves unresolved issues around privacy, security, and the unchecked expansion of AI into daily life, which these emerging technologies are set to accelerate. It can be argued, therefore, that without comprehensive safeguards, the integration of AI into personal and professional spaces may pose significant risks, e.g. data security and privacy, not to mention the risk of AI tools giving incorrect information or advice or displaying inbuilt bias towards the user they are supposed to be helping.
Six-Fingered Gloves
In a strange but related aside, a Finnish startup, Saidot, recently sent ominous six-fingered gloves to global tech leaders (including OpenAI’s Sam Altman) and EU politicians (and the UK Prime Minister) as a symbolic warning of AI dangers, particularly highlighting how image generators sometimes produce flawed outputs, like extra fingers. The gesture was aimed at raising awareness about the fast-evolving and unpredictable nature of AI, which could lead to unexpected consequences. Saidot’s CCO and co-founder, Veera Siivonen, said: “AI is developing so fast that nobody can fully anticipate its impacts and the emerging risks” and “That’s why we want to highlight both the steps that have been taken forward for safer AI, as well as some of the steps that should be taken.”
Saidot’s point aligns with the concerns surrounding the vetoed Senate Bill 1047, which sought to regulate AI technologies to prevent potential harm. As AI continues to develop rapidly, the failure to enact regulatory frameworks could leave many dangers inadequately managed.
What Does This Mean For Your Business?
The veto of Senate Bill 1047 by California State Governor Newsom highlights the need for achieving a delicate balance between promoting technological innovation and ensuring public safety. While the bill aimed to introduce necessary safeguards for advanced AI systems, its rejection shows the tension between regulation and the tech industry’s drive for unfettered progress. Newsom’s decision reflects the belief (particularly by the AI companies themselves) that overly rigid laws could stifle the rapid advancements in AI, which are viewed as essential for maintaining California’s competitive edge in the global tech landscape.
However, this move has also left a significant gap in AI governance. With AI systems becoming increasingly integrated into daily life (e.g. with the prospect of ‘ever-present’ AI as predicted by Microsoft), concerns about privacy, security, and potential misuse are mounting. The absence of comprehensive legislation leaves many of these issues unresolved, especially as the technology continues to evolve at an unprecedented pace. As argued by proponents of the bill, such as Senator Wiener, voluntary measures by AI companies may be insufficient and binding regulations are what’s really needed to protect society from potential harms, including cybersecurity risks and the creation of dangerous AI applications.
As AI continues to develop, the debate over how to effectively regulate it is far from over. The blocking of this bill may have slowed the momentum for immediate regulation, but it has also pushed the conversation forward. Looking ahead, policymakers, industry leaders, and experts will now need to collaborate on creating flexible yet effective frameworks that can both foster innovation and mitigate the risks associated with these powerful technologies.
For business users, the vetoing of Senate Bill 1047 and what would have been its wider effects means continued uncertainty around AI governance, leaving them reliant on voluntary safety measures from tech companies. While this may enable faster deployment of AI tools that enhance efficiency and innovation, for businesses it also raises risks. Without clear regulatory frameworks, businesses may face greater legal and ethical challenges, especially in areas like data security and AI accountability. For companies looking to integrate AI, the current absence of stringent safety measures could present both opportunities and risks as AI systems become more ingrained in business operations.
