AI Security Glossary

A technique that involves adding small, often imperceptible perturbations to input data to cause AI models to make incorrect predictions or classifications.

An attack where malicious actors gain control of autonomous AI agents, redirecting their actions to serve unintended purposes while maintaining the appearance of normal operation.

A type of attack where malicious functionality is embedded into an AI model during training, activated by specific trigger patterns in the input.

An attack technique that involves injecting malicious content into the context window of large language models to influence their responses or extract sensitive information.

The practice of intentionally corrupting training data to compromise the integrity and performance of machine learning models.

Synthetic media created using deep learning techniques to replace a person's likeness with someone else's, often used for deception or fraud.

A mathematical framework for quantifying and limiting the privacy loss when statistical information about a dataset is released.

A technique to reconstruct original data from learned embeddings or representations, potentially exposing sensitive information.

Attacks targeting federated learning systems where malicious participants can compromise the global model through poisoned local updates.

An attack where an AI agent's objectives are maliciously altered or redirected, causing it to pursue unintended goals while appearing to function normally.

When AI models, particularly language models, generate false or nonsensical information that appears plausible, potentially leading to misinformation.

Attacks that exploit the outputs or behavior of machine learning models to infer sensitive information about the training data or model parameters.

Techniques used to bypass safety measures and content filters in AI systems, particularly large language models, to generate prohibited content.

An attack that exploits the knowledge distillation process to extract information from teacher models or inject malicious knowledge into student models.

Techniques that modify the latent representations in generative models to control or manipulate the generated outputs in specific ways.

An attack that determines whether a specific data point was included in a model's training dataset by analyzing the model's behavior.

The process of stealing or replicating a machine learning model's functionality by querying it and training a substitute model on the responses.

A hidden functionality embedded in neural networks that can be activated by specific trigger patterns, causing the model to behave maliciously.

An attack technique where malicious instructions are embedded in prompts to manipulate large language models into performing unintended actions.

Attacks that exploit the quantization process used to compress neural networks, potentially introducing vulnerabilities or degrading performance.

A systematic approach to testing AI systems by simulating adversarial attacks to identify vulnerabilities and weaknesses before deployment.

A model trained to mimic the behavior of a target model, often used as a stepping stone for more sophisticated attacks like membership inference.

Attacks that attempt to recover specific training examples from machine learning models, potentially exposing sensitive or private information.

A single perturbation that can fool a neural network on most inputs from a given distribution, making it particularly dangerous for real-world attacks.

Techniques used to circumvent AI-based verification systems, such as biometric authentication or content verification mechanisms.

Techniques for embedding invisible markers in AI-generated content to enable detection and verification of synthetic media.

Attacks that work against AI models without requiring prior knowledge of the model's architecture, training data, or parameters.

An attack where malicious data is injected through API endpoints to corrupt AI model training or inference processes, often targeting real-time learning systems.

The phenomenon where AI systems amplify existing biases in training data, leading to discriminatory outcomes and unfair treatment of certain groups.

An attack technique that exploits the reasoning process of large language models by manipulating their step-by-step thinking to reach malicious conclusions.

A coordinated attack that overwhelms AI systems with computationally expensive queries, causing service degradation or complete failure.

Attacks that exploit unexpected behaviors that emerge from complex AI systems, particularly in multi-agent environments or large-scale deployments.

An attack where adversaries fine-tune pre-trained models on malicious data to introduce backdoors or alter model behavior while maintaining performance on benign tasks.

A privacy attack where sensitive information about training data is extracted by analyzing gradient updates in federated learning or distributed training scenarios.

Techniques to circumvent privacy-preserving computation methods that allow processing of encrypted data without decryption.

An attack that exploits the instruction-following capabilities of AI systems to make them perform unintended actions while appearing to follow legitimate commands.

An attack that corrupts knowledge graphs used by AI systems, introducing false relationships or entities to manipulate reasoning and decision-making.

A sophisticated backdoor attack that embeds triggers in the latent space of generative models, activated by specific patterns in the input representation.

Attacks that exploit vulnerabilities across multiple input modalities (text, image, audio) in multi-modal AI systems to achieve malicious objectives.

An attack that corrupts the neural architecture search process to produce models with hidden vulnerabilities or backdoors.

Attacks that alter the conceptual frameworks and taxonomies used by AI systems, leading to misclassification and reasoning errors.

A sophisticated attack technique that uses a sequence of carefully crafted prompts to gradually manipulate AI systems into performing prohibited actions.