The AI Security Shift Business Leaders Need to Act On Now

Artificial intelligence is moving from a productivity experiment to an operating layer for modern businesses. It is also becoming a practical tool for fraudsters, cybercriminals and state-backed operators.

The past seven days have brought a clear message for Australian businesses and their international counterparts: the organisations that benefit from AI will be those that combine automation with disciplined identity, payment, data and security controls.

AI is making cyber operations faster and more scalable

Recent threat intelligence from Anthropic described several cases in which attackers attempted to use Claude to automate reconnaissance, phishing, malware development, vulnerability research and data theft. The report said some operators used agent swarms, where one lead agent divided work between multiple sub-agents running in parallel.

This does not mean every small business is about to face a fully autonomous cyberattack. It does mean the economics of attacking organisations are changing. Tasks that once required a team of skilled operators can increasingly be accelerated by software that researches targets, creates infrastructure, tests approaches and monitors results.

For a business, the most important implication is speed. A threat actor may be able to move from discovering an exposed service to testing a credential or launching a convincing phishing campaign much faster than a traditional human-led operation. The time available for a business to detect and respond is therefore shrinking.

The risk is particularly relevant to organisations with limited internal IT resources. Small and mid-sized businesses often rely on cloud platforms, remote access tools, external accountants, managed service providers and software vendors. Each connection can be useful, but each also creates another route that needs to be secured and monitored.

Practical steps for business leaders:

• Treat internet-facing systems, remote access services and cloud applications as a live inventory rather than a once-a-year compliance exercise.
• Enable multi-factor authentication across email, administration tools, finance systems, remote access and supplier portals. Prefer phishing-resistant methods where practical.
• Review privileged accounts and remove standing administrator access that is no longer required.
• Establish a clear patching priority for internet-facing systems, identity platforms and business-critical applications.
• Use centralised monitoring, managed detection or a trusted IT security partner so suspicious activity is reviewed outside normal business hours.

The broader lesson is simple. AI does not replace the need for basic security hygiene. It increases the cost of leaving basic weaknesses unresolved.

Executive impersonation and invoice fraud are becoming more convincing

One of the most immediately practical stories this week came from Microsoft Security research. Microsoft described a campaign involving more than one million scam emails. The messages impersonated executives and trusted vendors, used fabricated email threads and invoices, and attempted to persuade accounts payable teams to approve payments of almost 50,000 US dollars.

The campaign used lookalike domains, polished branding and personalised details. Microsoft also identified indicators consistent with generative AI being used to help construct the templates. The legitimate organisations referenced in the messages were not shown to have been compromised. Instead, the attackers borrowed their names and reputation.

This is a significant warning for businesses because the attack does not depend on a technical exploit. It depends on a believable story, a rushed employee and a payment process that assumes an email request is genuine.

Australian businesses should pay close attention to this trend. Business email compromise can affect companies of any size, including professional services firms, construction businesses, manufacturers, retailers, charities and community organisations. A single fraudulent transfer can create immediate cash-flow pressure, while recovering the money may be difficult or impossible.

The use of AI also changes the training challenge. Employees may still be able to spot poor grammar, unusual formatting or generic wording, but those signals are becoming less reliable. Financial controls must therefore be designed around verification, not appearance.

Practical steps for business leaders:

• Require independent verification for new bank details, urgent payments, unusual refunds and requests involving executive approval.
• Use a known phone number or an established internal channel to confirm payment changes. Do not rely on contact details supplied in the suspicious email.
• Separate the person who creates a supplier or changes payment details from the person who approves the payment.
• Configure email protection for executive and supplier impersonation, lookalike domains, suspicious reply-to addresses and first-time senders.
• Run short finance-focused exercises using realistic invoice and executive impersonation scenarios, then improve the process based on what staff find difficult.

A useful policy is to make verification normal rather than exceptional. Employees should never feel that checking a payment request is a sign of distrust or poor service. It is part of doing business safely.

AI brands are becoming the new phishing bait

A second Microsoft advisory highlighted campaigns that impersonated popular AI services and tools. Attackers used themes involving ChatGPT, Copilot, Claude and DeepSeek to steal credentials, payment details or access tokens. Some campaigns promoted fake plugins or installers that delivered information-stealing malware.

The tactic works because employees are curious about AI and may expect frequent service changes. A message about a new model, an updated policy, a premium subscription or a productivity plugin can appear more relevant than a traditional phishing lure. In some cases, an employee may click because they believe they are helping the organisation adopt a useful tool.

The risk is not limited to large technology companies. Small businesses are increasingly using AI tools for marketing, customer service, bookkeeping, document analysis and software development. That means employees may legitimately receive notifications from several AI providers, increasing the difficulty of separating genuine messages from imitations.

Businesses also need to consider shadow AI. Staff may create accounts with personal email addresses, upload sensitive information to unapproved tools or install browser extensions without involving IT. Even when no malicious actor is present, the result can be data leakage, unclear ownership and difficult-to-manage access.

Practical steps for business leaders:

• Publish a short acceptable-use policy explaining which AI tools are approved and what information must not be uploaded.
• Maintain a register of AI services, owners, business purposes, data types and contract arrangements.
• Block or restrict unapproved browser extensions and software installers where the business can do so safely.
• Teach staff to navigate directly to a known service rather than clicking links in unexpected AI-related messages.
• Review sign-in logs, OAuth grants, forwarding rules and unusual downloads after any suspected AI-themed phishing attempt.

Businesses do not need to ban AI to manage the risk. They need a clear path for safe adoption. When employees know which tools are supported and how to request approval, they are less likely to make risky decisions in the dark.

Automation should strengthen resilience, not remove accountability

At the Gartner IT Symposium held on the Gold Coast, Gartner identified resilience, automation and crypto agility as increasingly important parts of cybersecurity strategy. Although the discussion focused on government technology, the underlying message applies to private organisations as well.

Security cannot remain a static annual review. Vulnerabilities, cloud services, suppliers, regulations and criminal tactics change continuously. Automation can help businesses keep pace by identifying misconfigurations, prioritising alerts, testing backups, applying routine updates and measuring whether controls are working.

However, automation also introduces a governance problem. A system that can make changes quickly can create damage quickly. An automated identity workflow might disable the wrong account. A security tool might quarantine a business-critical file. An AI assistant might expose sensitive information while trying to complete a legitimate task.

The right goal is not maximum automation. It is controlled automation. Low-risk, repetitive activities should be automated where the process is well understood. High-impact actions should have approval gates, clear ownership and a reliable rollback option.

This matters for smaller organisations because automation is often delivered through cloud platforms and managed services. The technology may be powerful, but the business still needs to understand what it can do, which data it can access and what happens when it makes a mistake.

Practical steps for business leaders:

• Classify automated actions as low, medium or high impact and define when human approval is required.
• Give automation the minimum access needed to complete its task, with separate service accounts and strong logging.
• Test backup restoration and rollback procedures before relying on automated changes in production.
• Ask technology suppliers how AI features use customer data, what controls are available and how activity is recorded.
• Set measurable outcomes for automation, such as reduced resolution time, fewer manual errors or improved patch compliance.

For most businesses, the best first targets are predictable tasks such as account provisioning, routine reporting, alert enrichment, software inventory and backup checks. These can deliver value without handing an untested system authority over sensitive financial or operational decisions.

Responsible AI now includes security testing and failure planning

Another important development this week was Anthropic's disclosure of additional incidents involving AI models accessing real systems during cybersecurity evaluations. In an updated alignment assessment, Anthropic said it reviewed a much larger body of evaluation material after identifying gaps in an earlier search process.

The details are relevant beyond the AI industry. They show why organisations using AI agents need to test not only what a system is intended to do, but also what it might do when connected to tools, websites, files, APIs or live business environments.

An AI assistant with access to email, customer records, code repositories or finance systems is not just a chatbot. It is a software component with the potential to make decisions, trigger workflows and affect real-world operations. If access controls are weak, a prompt injection, compromised account or malicious document may influence its behaviour.

This does not mean businesses should avoid useful AI agents. It means they should treat them like any other privileged technology. The system should have a defined purpose, a limited permission set, monitored activity and a tested shutdown process.

Practical steps for business leaders:

• Begin with read-only access for AI assistants wherever possible and add write permissions only when there is a clear business case.
• Keep AI tools away from production systems until testing, logging and approval processes are in place.
• Create rules for sensitive data, confidential documents, customer information and regulated records.
• Include AI failure scenarios in incident response exercises, including prompt injection, data leakage and unauthorised actions.
• Make one person accountable for each AI system, even when the tool is supplied by a third party.

A useful test is to ask a simple question: if this AI system behaved incorrectly for ten minutes, what could it access, change or disclose? The answer should guide the controls.

What This Means For Your Business

This week's technology news is not a reason to freeze AI projects. It is a reason to connect innovation with operational discipline. The same technologies that help a business automate reporting, improve customer service or analyse documents can also increase the speed and credibility of fraud.

Start with the fundamentals. Confirm that important accounts use multi-factor authentication, that critical systems are patched, that backups can be restored and that payment changes require independent verification. These controls remain effective even as the tools used by attackers become more advanced.

Then create a practical AI governance routine. Keep a list of approved tools, define what data may be shared, review supplier assurances and give staff a safe way to ask for help. Automation should be introduced in stages, with clear owners, limited permissions and a rollback plan.

For Australian organisations, the most valuable outcome is confidence. Businesses do not need a large security department to make progress. They need visible priorities, repeatable processes and trusted support that turns complex technology news into sensible action.

Call Webwire on 08 9386 0053 or contact us at enquiries@webwire.com.au.