Ensuring security in AI copilots: tackling the evolving threat landscape

AI copilots are rapidly becoming the operational nerve centre for many organisations, but their usefulness brings concentrated security risk across the entire data pipeline. According to the original presentation by Andra Lezza, Principal AppSec Specialist, the core challenge is protecting sensitive material , intellectual property, financial records and personally identifiable information , from ingestion through monitoring, because persistent access or leakage of that data creates major liability for firms. ^[1]

Many threats are familiar to application-security teams, but they take new forms in LLM-based assistants. Traditional failures such as broken access control, misconfiguration and insecure third‑party components still dominate: industry guidance stresses that roughly 85% of AI security is “traditional security done well” adapted to new interfaces like prompts, plugins and embedding services. At the same time, generative models introduce novel risks , system prompt leakage, vector/embedding weaknesses in retrieval-augmented generation, and intrinsic misinformation or hallucination , that demand fresh controls. ^[1][6]

Two deployment patterns illustrate the trade-offs. An independent copilot is tightly embedded in a single product and can offer deep domain logic and faster responses, but deeper access to specific datasets increases the potential impact of targeted attacks such as data poisoning or cached-data disclosure. An integrated, multi‑tenant copilot scales across products with shared services and generic skills, but creates a complex permission matrix and a larger attack surface for cross‑tenant data leakage and lateral movement. Lezza emphasises there is no inherently safer architecture , security depends on implementation. ^[1]

Prompt injection remains a particularly intractable problem: a user-supplied input can alter an assistant’s behaviour, exfiltrate system prompts or cause the model to request privileged actions. Mitigations combine prompt engineering, runtime guardrails and templated job definitions that limit the model’s view of data and the actions it may request. Lezza recommends a defence‑in‑depth approach that validates both inputs and outputs and funnels LLM interactions through constrained templates and skills. Industry guidance and vendor tools (for example, cloud providers’ built‑in guardrails) should be used where appropriate, but researchers continue to find weaknesses in real‑world agents. ^[1][3][4]

Supply‑chain threats extend from CI/CD tool compromise to poisoned training data and malicious models. The presentation highlights controls such as model‑hash verification, safetensors or other safe file formats, behavioural testing and AI “red‑teaming” to detect backdoors or harmful outputs before deployment. Recent academic work has formalised evaluation frameworks for agent security , for example, ASTRA tests system‑prompt level guardrails and autonomous agent behaviour against attacks inspired by the OWASP LLM Top Ten, demonstrating wide variance between models’ ability to respect boundaries. ^[1][2]

Operational controls map to the five‑stage AI pipeline Lezza outlines: secure ingestion (encryption, provenance and automated compliance checks); transformation and training (adversarial training, differential privacy); deployment (immutable artifacts, signed containers, SBOMs); and monitoring (real‑time auditing, continuous feedback into retraining). DevSecOps practices , SAST/DAST, SBOMs, Sigstore signing, ephemeral training environments , remain essential to reduce the risk surface around model artefacts and dependencies. ^[1][6]

Access control must be granular and derived from user tokens and session context; Lezza describes On‑Behalf‑Of scoped tokens and recommends deriving the copilot’s least‑privilege scope from the requesting user’s permissions. Organisations should consider ACLs, RBAC or ABAC as fits their environment, plus tenant segregation, rate limiting and strict auditing to limit financial and availability abuse (for instance, token‑draining floods of LLM requests). Several practitioner guides reinforce these mitigations and the need for proactive governance of third‑party integrations. ^[1][4][5]

Because many of the OWASP LLM Top Ten and practitioner blogs converge on similar mitigations, a practical mitigation programme is to: (1) adopt templates and guardrails that strictly confine the model’s visibility and actions; (2) vet and sign models and dependencies; (3) perform continuous model behaviour testing and red‑teaming; and (4) instrument detailed logging and provenance to detect caching, leakage or pivoting attempts. Active monitoring and human‑in‑the‑loop gates, together with quantified risk‑assessment workflows, are essential as the landscape and regulatory expectations evolve. ^{[3][4][5][6][7]}

Securing AI assistants is therefore an extension of established secure‑engineering practice adapted for the particularities of LLMs: generative failure modes, embedding vectors and agentic capabilities. The practical advice is defence in depth, granular least‑privilege controls, continuous testing (including adversarial scenarios) and tight supply‑chain hygiene , supplemented by the latest evaluation frameworks and threat models so organisations can validate that their guardrails actually hold in the face of novel attacks. ^[1][2][3][6]

📌 Reference Map:

##Reference Map:

• ^[1] (InfoQ transcript of Andra Lezza presentation) - Paragraph 1, Paragraph 2, Paragraph 3, Paragraph 4, Paragraph 5, Paragraph 6, Paragraph 7, Paragraph 8, Paragraph 9
• ^[2] (arXiv ASTRA paper) - Paragraph 5, Paragraph 9
• ^[3] (ActiveFence blog) - Paragraph 4, Paragraph 8, Paragraph 9
• ^[4] (Protecto.ai blog) - Paragraph 4, Paragraph 6, Paragraph 8
• ^[5] (Medium: OWASP LLM Top 10) - Paragraph 8
• ^[6] (OWASP GenAI data security white paper) - Paragraph 2, Paragraph 6, Paragraph 9
• ^[7] (Bugcrowd blog) - Paragraph 8

Source: Noah Wire Services