Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications IV

Passive sonar signals contain complex characteristics often arising from environmental noise, vessel machinery, and propagation effects. While convolutional neural networks (CNNs) perform well on passive sonar classification tasks, they can struggle with statistical variations that occur in the data. To investigate this limitation, synthetic underwater acoustic datasets are generated that centered on amplitude and period variations. Two metrics are proposed to quantify and validate these characteristics in the context of statistical and structural texture for passive sonar. These measures are applied to real-world passive sonar datasets to assess texture information in the signals and correlate the performances of the models. Results show that CNNs underperform on statistically textured signals, but incorporating explicit statistical texture modeling yields consistent improvements. These findings highlight the importance of quantifying texture information for passive sonar classification.

Large representative datasets are essential for developing and testing tracking and detection algorithms. However, acquiring real-world data across a desired target-feature trade space can be challenging. Synthetic data generation can address these challenges if it provides radiometric fidelity, flexibility in sensor, target, and scene attributes, realistic dynamic conditions, and relevant noise sources. This paper presents a toolkit for generating synthetic high-speed video that meets these criteria. We describe our approach through a case study simulating an infrared sensor in geostationary orbit observing dynamic targets against an Earth background. The toolkit's capabilities are assessed by comparing simulation results with first-principles radiometric calculations and validating details such as target motion, signal-to-noise ratio (SNR), and scene clutter. Our toolkit fulfills a community need by generating realistic high-speed video, enabling rapid development and testing of algorithms while reducing costs and lead times associated with acquiring real-world data.

Advancing autonomous perception for defense applications requires AI models capable of reliable object detection in complex, dynamic battlefield environments. Achieving this performance depends on diverse, high-quality datasets that reflect real-world sensing conditions, which are often impractical to collect physically. This paper introduces a physics-based virtual simulation framework for generating synthetic, multimodal datasets tailored to battlefield scenarios. The framework combines high-fidelity 3D CAD models with configurable virtual sensors to simulate optical, LiDAR, and Synthetic Aperture Radar (SAR) modalities under varied operational conditions. It supports controlled variation of sensor range, geometry, and illumination to ensure balanced, comparable datasets across modalities. An integrated annotation module provides automated polygon labeling and adaptive sampling to enhance class balance and reduce redundancy. Synthetic LiDAR includes elevation, reflectivity, and noise modeling to emulate realistic behavior. The resulting datasets capture diverse tactical scenes and enable scalable, reproducible training and evaluation of AI-based object detection systems.

Simulation environments for AI demand structured, error resilient generation pipelines. In our previous work, we introduced LSCENE and LCAP, two formal languages designed to describe virtual worlds and data collection processes for AI/ML. In this article, we extend that work by introducing role based agents that operate in parallel. To mitigate agentic issues, each agent is grounded in an independently typed vocabulary and is equipped with reflection access to Unreal Engine methods and assets. Overall, the division of labor when constructing a scene helps reduce each agent's individual context, task complexity, and time spent working. To evaluate the effectiveness of the proposed approach, we conducted a series of experiments. The first demonstrates the benefits of incorporating reflection. The second examines agent design considerations, including the importance and nuances of task specification, role, and structured output. The final experiment demonstrates the potential of our framework for constructing complex datasets to train and evaluate computer vision algorithms for drones.

Identifying, classifying, and monitoring space launch vehicles is vital to Space Domain Awareness (SDA) and supports defensive decisions that protect critical assets in congested and contested orbital environments. This work introduces a physics-driven methodology for generating multimodal synthetic datasets that integrate vehicle dynamics, material properties, environmental conditions, and correlated EO/IR, LiDAR, and SAR signatures. The resulting observables capture formations, spatial extent, and geolocation characteristics across diverse sensor viewpoints, providing the variability needed to train AI systems. By simulating a wide range of operational scenarios and vehicle configurations, the dataset enables ML models to generalize beyond empirical baselines and adapt to dynamic, non-peacetime conditions. Applicable to ground-based and aerial sensing platforms, the approach supports predictive assessment of object behavior, system readiness, and operational performance. For defense applications, it improves mission planning, sensor tasking, and object custody while revealing observational gaps and informing architecture design to enhance resilience in modern space operations.

Deep learning has proven effective for detecting and identifying embedded targets in image-based scenes, but high performance requires a large, diverse dataset for robust algorithm training. Obtaining sufficient measured data of adversarial assets can be particularly challenging in the thermal infrared wavebands, but an accurate synthetic image-generation methodology can supplement available measured imagery. In this study, synthetic images of UAVs are generated using physics-based MuSES infrared simulations for a variety of weather conditions, times of day, sensor and target positions. This automated process for generating comprehensive datasets can incorporate motion blur of spinning propeller blades and insert synthetic targets into measured background scenes. The performance of YOLO11 and Faster-RCNN algorithms are evaluated using sky backgrounds for various target resolutions to assess effectiveness in detecting and recognizing UAVs in the thermal infrared spectrum. Detection and identification accuracy are examined for measured and synthetic test images for several sensor-target ranges.

Computer vision techniques for articulating 3D objects are dependent on learning from large amounts of training data that characterize the full range of object types, appearance, pose relative to the camera, and deformations. To overcome this dependence on exhaustive 2D training sets, we propose a zero-shot 4D template matching approach that learns instantaneous 3D structure from static images and evaluates the likelihood of a temporal deformation model. We fit Hidden Markov Models to a model library of synthetic, 3D animations of aerial objects (bird, drone, parachute, helicopter, flag), and compute the likelihood that a given 4D input fits the process. Using this approach, we demonstrate reliable classification across two domains: synthetic 4D inputs outside the model library and 4D inputs reconstructed from real RGB-D video sequences. Overall, this work establishes an unsupervised, zero-shot approach to non-rigid, aerial object detection that is both effective in practice and inherently extensible, scaling seamlessly to new object types through the simple addition of synthetic animations to the model library.

The use of synthetic imagery to train AI/ML algorithms continues to play a significant role in supplementing datasets to improve algorithm performance. AI/ML algorithms trained with synthetic data suffer from the synthetic-to-real domain gap, in which performance of algorithms trained solely with synthetic data do not perform as well as algorithms trained with real data. This work explores the use of network dissection to identify concept differences between synthetic and real imagery to inform modifications of synthetic imagery to reduce the domain gap.

Credit card fraud exceeded $10 billion in U.S. losses in 2023, yet fewer than 5% of transactions carry verified labels. The Gated Temporal Attention Network (GTAN, Xiang et al. AAAI-23) addresses class imbalance via semi-supervised label propagation over temporal transaction graphs. Three sampling strategies are evaluated on IEEE-CIS, benchmarked against S-FFSD: stride-based downsampling, front-loaded fraud concentration, and pre-context window sampling. Data partitioning uses a standard stratified split; temporal ordering is enforced separately during graph construction. Threshold sweeps show sampling strategy determines score calibration: stride-based downsampling sustains high precision across a wide threshold range, while front-loaded and pre-context strategies yield lower ceilings. Pre-context window sampling inflects at a lower fraud concentration than front-loading, supporting the hypothesis that authentic temporal predecessors strengthen risk propagation. Metrics are fold-order dependent due to GTAN's single-checkpoint evaluation, requiring multi-seed averaging for robustness. Zero-shot cross-dataset transfer yields highly unstable recall due to embedding reinitialization.

Dr. Manser will discuss her work as Subject Matter Expert in Synthetic Data for the Army's DEVCOM C5ISR Center, focusing on applications for Defense in Computer Vision and related AI/ML tasks. The talk centers on the complexity of synthetic data generation, efforts to improve fidelity and performance, and a diverse array of current applications spaces.

Infrared (IR) detection and tracking technologies are advancing rapidly with the integration of artificial intelligence and machine learning (AI/ML). While these tools enhance capability and automation, they also present challenges in ensuring consistent and transparent system performance across diverse operational conditions. Reliable AI/ML training depends on large, high-quality datasets, yet real IR data are often limited. Synthetic data offers a promising solution, but their influence on model performance and trustworthiness remains uncertain. This work compares AI object detectors trained using the empirical ATR Algorithm Development Image Database (ADID) and a synthetic counterpart developed by Perez, Vanstone, et. al. Leveraging explainable AI (XAI) tools the study evaluates how synthetic data may affect model decisions. Results contribute to a framework for validating AI/ML systems that use synthetic data, promoting greater transparency, reliability, and confidence in future IR sensing applications.

The naval ship infrared signature model and naval threat countermeasure simulator (ShipIR/NTCS) developed by W.R. Davis Engineering Ltd. has undergone extensive validation since its adoption as a NATO-standard and through US Navy Accreditation for Live Fire Test and Evaluation of the DDG-79 (Flight IIA) Aegis Guided Missile Destroyer and Contract Design of the DDG-1000 (USS Zumwalt) Guided Missile Destroyer. Some of the features that make this tool well suited for input to AI / ML applications is its fully-deterministic approach to thermal / EOIR modelling. With the delivery of a turn-key ShipIR thermal and EOIR model of each class of ship, only a few climatic and ship operational inputs are required to construct a full operating EOIR scenario with both transient background and ship signature conditions.This paper will describe the overall framework but also its scene rendering and scenario model updating capabilities. It is hoped that new use case evaluations might result from the presentation of these details to a wider Artificial Intelligence and Machine Learning Community.

Modern automated target recognition (ATR) systems can use a diverse set of modalities to identify targets. Of particular importance are infrared emissions for night-time detection. In this paper, we describe a physically implemented infrared camouflage system that uses a limited number of temperature-adjustable panels that can be placed on a vehicle and adversarially trained to evade ATR by changing the infrared emitted by the panels. Testing with a YOLO ATR with and without the camouflage activated, we consistently degrade detection from 90% to below 52% confidence. This was achievable even when only activating 14 of the 24 available panels. This improves upon previous work by creating a cost-effective solution that is optimized to avoid ATR instead of attempting to modify the infrared pattern of the entire target.

Adverse weather, such as heavy fog, rain, or snowstorm reduces the performance of object detection in surveillance systems. Collecting diverse real‑world data for these rare events is difficult, limiting model robustness. To address this, we propose a generative‑AI‑based coherent data augmentation framework that synthetically simulates realistic weather effects on surveillance footage. Using image‑to‑video generation, prompt engineering, and cascading pipelines, our method produces photorealistic data for benchmarking and developing detectors capable of sustaining accuracy under challenging environmental conditions.

This study explores using diffusion-based synthetic data to improve military vehicle detection in low-data environments. We fine-tuned FLUX.1 [dev] via LoRA using only 8 or 24 images per class to generate training samples for an RF-DETR detector. Results show synthetic data significantly boosts performance, yielding up to +8.0% mAP50 in the 8-sample regime. Additionally, incorporating structural guidance via ControlNet provided further gains (+4.1% mAP50) under extreme scarcity. Our findings demonstrate that object-specific generative models offer a powerful, data-efficient alternative to traditional simulation pipelines for training military AI systems when real-world data is limited.

This work presents an indoor proving ground framework for training and evaluating air-to-ground vehicle detection models. To avoid costly outdoor acquisition campaigns, we reuse miniature military vehicle models—previously employed for semi-synthetic training—within a physically realistic diorama that simulates diverse environments and conditions such as occlusions, clutter, smoke, and camouflage. Images are captured using real camera systems, ensuring realistic sensor characteristics. By employing identical miniature vehicles in both training and testing, the approach reduces the domain gap between synthetic and real data, enabling focused investigation of model performance under complex scenes and between visually similar military vehicles. The framework provides a cost-effective, scalable solution for developing robust AI-based vehicle detection, recognition and identification.

This work evaluates the efficacy of modern detection techniques against diffusion-generated satellite imagery in restrictive battlefield conditions. This approach evaluates common open-source techniques such as comparing semantic embeddings and evaluating image residual image noise, as well as a novel hybrid classification framework that combines semantic and forensic features. Real satellite images are compared against Stable Diffusion 3.5–generated imagery subjected to realistic levels of expected degradation from corruption, compression, and other negative artifacts common to the battlefield. This work also evaluates the resource burden inherent in each of the evaluated techniques to help quantize the feasibility of each in a constrained environment. Results show that semantic features dominate detection performance, while traditional PRNU methods largely fail on diffusion imagery. The hybrid approach improves calibration and confidence, highlighting limitations of classical forensics and the need for multi-modal detection strategies in ISR contexts.

Reliable electro-optical UAS sensing is challenged by low-pixels-on-target imagery and confusion between UAS and avian species, where operationally important regimes are sparsely represented in labeled data. This paper describes methodological building blocks for a closed-loop synthetic data generation (SDG) capability, focusing on feedback signals that prioritize which synthetic scenarios to generate next. We combine (i) regime-conditioned performance analysis using metadata-driven grouping and quantile binning over SDG parameters (e.g., range and location) with (ii) ensemble-based uncertainty quantification to identify parameter regions associated with elevated error and uncertainty. We further include a simulation-to-real fidelity benchmark using paired real--synthetic replicas and image similarity metrics (SSIM, CW-SSIM, FSIM) to contextualize synthetic-data quality and expected transfer limitations. The resulting framework provides a practical strategy for selecting targeted synthetic augmentations and defining hypotheses to be tested in future end-to-end closed-loop iterations. SNL is managed and operated by NTESS under DOE NNSA contract DE-NA0003525

The human visual system takes only seconds to learn to identify new or modified objects, but it takes computer vision models months to years to do the same thing. Synthetic data promises to resolve this difference, but currently it still takes months to generate useful synthetic datasets even with significant amounts of compute. This time delay and resource requirement means that military units can not adapt their AI targeting and autonomous systems at the edge. We demonstrate that using our fully open source SIMPL data generation method that existing overhead detection and classification models for computer vision tasks can be fine tuned to adapt to and/or recognize novel targets in under 24 hours with minimal compute. This period includes generation of training data and model training/testing. This revolutionary jump in speed enables military units to adapt their targeting systems in the field and at the speed of relevance enabling them to remain effective in the face of a constantly changing adversary. We provide an ablation on the performance achievable in a 24-96 hour adaption period (e.g., standard military mission planning periods) based on available compute hardware: CPU

Optical turbulence degrades free-space optical systems and is costly to model accurately. Here, we present TurPy, a high-fidelity, differentiable wave optics simulator for generating synthetic turbulence data. Using a memory-efficient split-step autoregressive algorithm with compressed phase screens, TurPy captures key atmospheric effects like wind, convection, and temporal correlation. It connects seamlessly to deep learning workflows to train downstream algorithms as well as enabling end-to-end optimization of system parameters via gradient-based methods. Validated against theoretical models with over 95% accuracy, TurPy also automates phase screen placement and demonstrates preliminary wavefront optimization. This tool bridges physics-based simulation and AI to enhance data generation, system design, and optical performance under turbulence.

Inferring initial parameters from an output is a major challenge for "black-box" generative models, as a single output can map to multiple valid inputs. We introduce a novel framework using a conditional denoising diffusion model to predict a multimodal distribution of rendering parameters from a 2D image. By analyzing visual cues like shadows and perspective, our model learns the complex mapping between an image and its source parameters. This approach allows us to generate a diverse, yet accurate range of plausible parameter sets from a single image, effectively capturing the problem's inherent ambiguity. The method is demonstrated by inferring camera and lighting settings used to render a 3D vehicle.