arXiv Computer Vision

PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?

arXiv Machine Learning
Aug 20

A Comprehensive Review of Large Language Models for Nanophotonics: From Surrogate Modeling to Autonomous Design

The review examines how Large Language Models (LLMs) enhance nanophotonics design by providing semantic interfaces, code generation, and tool orchestration. It traces the evolution from classical neural networks to transformer-based models and categorizes LLM applications into surrogate models that map structure to spectrum and agentic systems that generate code and orchestrate simulations for closed-loop optimization. The article also highlights potential cross-disciplinary uses of LLMs in materials science and wireless communications, and envisions future multimodal foundation models that actively collaborate in autonomous scientific discovery.

By Huanshu Zhang, Kegeng Tang, Lei Kang, Sawyer D. Campbell, Zihao Wang, Douglas H. Werner
arXiv AI
Aug 24

VortexChat: An agentic framework for autonomous multi-objective integrated photonic design

VortexChat is an agentic framework that autonomously performs end-to-end inverse design of integrated photonic devices from natural language specifications. It combines a large language model decision agent with topology generation, gradient-based refinement, and full-wave electromagnetic simulation in a closed-loop architecture, enabling iterative decomposition of design objectives and minimal human intervention. The system successfully generated devices meeting the Vortex100 Benchmark metrics and fabricated a broadband terahertz perfect vortex beam multiplexer that matched simulation predictions.

By Faqian Chong, Yulun Wu, Shilong Li, Andrew Forbes, Hongsheng Chen, Song Han
arXiv AI
Aug 28

PICasso: An AI-Enabled Design Framework for Autonomous Optimization of Silicon Photonic Devices

PICasso is an AI‑enabled framework that converts natural‑language specifications into manufacturable silicon photonic integrated circuits (PICs) through a structured pipeline of NL → YAML → GDS, PDK‑aware knowledge injection, automated placement and routing, DRC/LVS validation, and SAX‑based photonic simulation. The authors introduce PIC‑Set, a benchmark of 36 parameterized PIC design tasks, and evaluate several large language models (LLMs) using new metrics such as structural and functional Spec@k, optimization efficiency, and robustness. Across the benchmark, PICasso markedly improves specification satisfaction, achieving up to 92.7% structural Spec@3 and 52% functional Spec@3, while reducing mean insertion loss from 4.98 dB to 3.25 dB through simulation‑guided optimization.

By Deepak Vungarala, Deniz Najafi, Abdulrahman Aljoudi, Zahra Ghanaatian, Navid Khoshavi, Gourav Datta, Arman Roohi, Mahdi Nikdast, Shaahin Angizi
arXiv AI
Jul 13

A Self-Evolving Agentic Framework for Metasurface Inverse Design

arXiv:2604. 01480v2 Announce Type: replace Abstract: Metasurface inverse design can realize complex optical functionality, but turning a target optical response into executable optimization code still requires substantial expertise in computational electromagnetics and solver-specific software engineering.

By Yi Huang, Bowen Zheng, Yunxi Dong, Hong Tang, Huan Zhao, S. M. Rakibul Hasan Shawon, Hualiang Zhang
arXiv Computer Vision
Sep 28

LensDesigner: A Self-Improving Agent for Optical Lens Design

LensDesigner is an autonomous agent framework that emulates expert opticians to tackle the complex, non‑convex problem of optical lens design. It uses a large lens library (LensLib100K) and optics‑aware retrieval to provide valid structural seeds, then iteratively improves through a curriculum agent that learns and reuses design heuristics. The system is evaluated on LensArena, a benchmark of 120 diverse optical tasks, where it outperforms existing baseline algorithms in success rate and optimization efficiency.

By Lei Sun, Haoran Liang, Dannong Xu, Yao Gao, Yuyu Geng, Jinjin Gu, Kaiwei Wang, Danda Pani Paudel, Luc Van Gool
arXiv AI
Sep 18

VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizing

The paper introduces VLM-CAD, a workflow that uses Vision Language Models (VLMs) for analog circuit sizing while mitigating spatial blindness and logical hallucinations. It incorporates a neuro‑symbolic parsing module, Image2Net, to convert schematics into topological graphs and JSON, and an Explainable Trust Region Bayesian Optimization method, ExTuRBO, to guide design decisions with sensitivity evidence. Experiments on 12 sizing tasks across six circuits and four technology platforms show a Strict Pass@1 of 23.3% and a Relaxed Pass@1 of 91.7%.

By Guanyuan Pan, Shuai Wang, Yugui Lin, Tiansheng Zhou, Pietro Li\`o, Zhenxin Zhao, Yaqi Wang
arXiv AI
Jul 9

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607. 07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging.

By Ethan Chung, Chuanjun Zheng, Jasper Tan, Jingxi Li, Haopeng Zhang, Huaijin Chen
arXiv Machine Learning
Jul 23

Q-PhotoNAS: Hybrid Quantum Neural Architecture Search Framework on Photonic Devices

arXiv:2605. 22097v2 Announce Type: replace-cross Abstract: Photonic quantum computing is a promising platform for scalable quantum machine learning, but designing effective hybrid architectures remains challenging under hardware and optimization constraints.

By Farah Elnakhal, Alberto Marchisio, Nouhaila Innan, Gabriel Falcao, Muhammad Shafique
arXiv Machine Learning
Aug 28

Towards a universal meta-optics solver via large language models

The paper introduces a unified large language model workflow for modeling and inverse-design of metasurfaces across multiple families. By converting geometries, design parameters, and optical responses into a shared instruction‑following text format, the authors fine‑tune Gemma‑2‑9B on eight distinct metasurface families. Compared to single‑family models, the joint model predicts all families’ optical responses simultaneously and reduces mean‑squared error by an average of 56.5%. "whyItMatters":"The approach demonstrates that a shared sequence‑based LLM interface can streamline cross‑family metasurface design, eliminating the need for separate surrogate architectures for each geometry class."

By Huanshu Zhang, Lei Kang, Yuyan Chen, Luxiang Wang, Zhaolong Cao, Douglas H. Werner