End-to-end optimization environment built from a differentiable ray tracer

Advances in computational imaging increasingly demand a design loop in which optical systems and post capture image processing algorithms are conceived and optimized together. We present a co-design framework built on the differentiable ray tracing software FORMIDABLE, a Julia based tool originally developed at ONERA under ESCL licence. FORMIDABLE provides accurate, differentiable simulations of complex optical systems, allowing easy evaluation of system performance metrics (e.g., modulation transfer function, spot diagrams, etc.) within an optimization pipeline. We extend this framework to incorporate image reconstruction based on Wiener filters, allowing simultaneous optimization of the optical design and the associated image processing algorithm. The extended framework includes the optimization on metrics computed after processing, such as mean squared error, both with Infinite Impulse Response and Finite Impulse Response Wiener filters. We present an example where this framework is used to jointly optimize a Petzval aspheric system used with a single Wiener deconvolution, with the goal of extending the Depth of Focus while retaining satisfactory performance across the whole Field of View. For this purpose, we use cost functions tailored for Depth of Focus extension, that combine co-design metrics evaluated for multiple Field of View points and throughout defocus. We show that a pre-optimization on MTF equalization is necessary to increase system performance in both the Field of View and the Depth of Focus, and that the size of the deconvolution filter needs to be large enough when compared to the point spread functions of the optical system.