Executing algorithms on future quantum information processing devices will rely on the ability to continuously monitor the device's state via quantum measurements and to act back on it, on timescales much shorter than the coherence time, conditioned on prior observations. Such real-time feedback control of quantum systems, which offers applications e.g. in qubit initialization, gate teleportation and quantum error correction, typically relies on an accurate model of the underlying system dynamics. With the increasing number of constituent elements in quantum processors such accurate models are in many cases not available. In other cases, obtaining an accurate model will require significant theoretical and experimental effort. Model-free reinforcement learning promises to overcome such limitations by learning feedback-control strategies without prior knowledge of the quantum system. Reinforcement learning has had success in tasks ranging from board games to robotics. Reinforcement learning has only very recently been started to be applied to complex physical systems, with training performed either on simulations or directly in experiments, for example in laser, particle, soft-matter and quantum physics. Specifically in the quantum domain, during the past few years, a number of theoretical works have pointed out the great promises of reinforcement learning for tasks covering state preparation, gate design, error correction, and circuit optimization/compilation, making it an important part of the machine learning toolbox for quantum technologies. In first applications to quantum systems, reinforcement learning was experimentally deployed, but training was mostly performed based on simulations, specifically to optimize pulse sequences for the quantum control of atoms and spins. Beyond that, there are two pioneering works demonstrating the training directly on experiments which was used to optimize pulses for quantum gates and to accelerate the tune-up of quantum dot devices. However, none of these experiments featured real-time quantum feedback. Real-time quantum feedback is crucial for applications like fault-tolerant quantum computing. Realizing it using deep reinforcement learning in an experiment has remained an important open challenge. Very recently, a step into this direction was made in ref., which demonstrates the use of reinforcement learning for quantum error correction. In contrast to what we present in this paper, these experiments relied on searching for the optimal parameters of a controller with fixed structure. Here, we realize a reinforcement learning agent that interacts with a quantum system on a sub-microsecond timescale. This rapid response time enables the use of the agent for real-time quantum feedback control. We implement the agent using a low-latency neural network architecture, which processes data concurrently with its acquisition, on a field-programmable gate array (FPGA). As a proof of concept, we train the agent using model-free reinforcement learning to initialize a superconducting qubit into its ground state without relying on a prior model of the quantum system. The training is performed directly on the experiment, i.e., by acquiring experimental data with updated neural network parameters in every training step. In repeated cycles, the trained agent acquires measurement data, processes it and applies pre-calibrated pulses to the qubit conditioned on the measurement outcome until the agent terminates the initialization process. We study the performance of the agent during training and demonstrate convergence in less than three minutes wall clock time, after training on less than 30,000 episodes. Furthermore, we explore the strategies of the agent in more complex scenarios, i.e. when performing weak measurements or when resetting a qutrit.

The paper extensively reviews the existing literature on quantum feedback control and reinforcement learning applied to quantum systems. It highlights the limitations of model-based approaches for increasingly complex quantum processors and emphasizes the potential of model-free reinforcement learning to overcome these limitations. The authors cite numerous works demonstrating the application of reinforcement learning in various fields, including board games and robotics, and specifically in the context of physical system control. The literature review focuses on the recent surge in applying reinforcement learning to complex physical systems, both through simulations and direct experimental implementation. Existing works on reinforcement learning in quantum physics are discussed, emphasizing those that address tasks such as state preparation, gate design, error correction, and circuit optimization. The authors clearly delineate the gap in the literature, noting the lack of experimental demonstrations of real-time quantum feedback using deep reinforcement learning, despite theoretical predictions of its potential. The paper positions its work as addressing this crucial gap and pushing the boundaries of real-time quantum control.

The researchers employed model-free reinforcement learning to develop a real-time quantum feedback control agent. The agent, implemented as a low-latency neural network on a field-programmable gate array (FPGA), interacts with a superconducting transmon qubit. The experimental setup involved continuous monitoring of the qubit's state via quantum measurements. The agent's actions consisted of applying pre-calibrated pulses to the qubit, conditioned on the measurement outcomes. The observations were obtained through microwave scattering off a coupled resonator, digitized into I and Q components. These components formed the observation vector, which the agent used to select actions based on its policy (a conditional probability distribution modeled as a neural network). The agent's policy is a neural network with parameters θ. The goal was to maximize the cumulative reward R, which was a function of the speed and fidelity of initializing the qubit into its ground state. The reward function was designed to balance initialization fidelity and the number of cycles needed to reach the target state, controlled by a parameter λ. The training process involved transferring batches of episodes to a personal computer (PC), where the rewards were computed and the network parameters were updated using a policy-gradient method. The updated parameters were then returned to the FPGA for real-time operation. The FPGA implementation featured a unique network architecture designed to minimize latency by processing new data concurrently with its acquisition. The authors meticulously analyzed the agent's performance by monitoring the average number of cycles until termination and the initialization error. They also explored the agent's strategies in various scenarios, including strong and weak measurements, and qutrit reset, to demonstrate the agent's adaptability and effectiveness in complex situations.

The study successfully demonstrated a real-time reinforcement learning agent for quantum feedback control with sub-microsecond latency. The agent, implemented on an FPGA, learned to initialize a superconducting qubit to its ground state with high fidelity (error <0.2%) within three minutes of training on less than 30,000 episodes. The agent’s strategy was analyzed, revealing that it closely followed optimal thresholding strategies in high-certainty regimes but smoothly transitioned in low-certainty regimes, suggesting that it utilized more information than just the integrated signal. The performance was benchmarked against a simpler threshold-based approach, showing comparable results in strong measurement scenarios. Experiments with weak measurements demonstrated the agent's ability to leverage memory from previous measurement cycles to improve initialization speed and fidelity. Moreover, extending the agent to handle qutrit states (three energy levels) showed its capacity to effectively reset the system even when considering unwanted leakage to a higher excited state, showcasing the versatility of the developed method. The agent’s ability to adapt to variations in measurement strength and the number of states considered highlights its robustness and general applicability to different experimental settings. The overall results confirm the efficacy of reinforcement learning in creating efficient real-time quantum feedback protocols, paving the way for its broader application in quantum information processing.

The successful implementation of a sub-microsecond latency reinforcement learning agent for real-time quantum feedback represents a significant advancement in quantum control. The ability to initialize a qubit to its ground state with high fidelity using a model-free approach overcomes the limitations of model-based techniques, which become increasingly challenging as quantum systems grow in size and complexity. The agent’s performance, comparable to or exceeding simple optimal strategies, validates the effectiveness of reinforcement learning in discovering efficient control protocols. The results have broader implications for quantum computation, as the real-time feedback is crucial for fault tolerance and error correction. The adaptability demonstrated through experiments with weak measurements and qutrit control showcases the flexibility of the method and its potential for handling more complex and noisy quantum systems. The fast training time enables the agent to adapt to experimental drifts, which enhances the system’s robustness. This research opens avenues for further exploration of reinforcement learning in a variety of quantum feedback tasks, pushing the boundaries of real-time quantum control and paving the way for more complex quantum technologies.

This research successfully implemented and trained a real-time reinforcement learning agent capable of sub-microsecond latency control of a superconducting qubit. The agent efficiently initialized the qubit to its ground state with high fidelity and demonstrated adaptability across various scenarios, including weak measurements and qutrit control. This work significantly advances real-time quantum feedback control, overcoming limitations of model-based methods. Future research directions include scaling the approach to multi-qubit systems, investigating more complex reward functions, and exploring applications in quantum error correction and many-body feedback cooling. The method's adaptability and efficiency hold promise for enabling the development of more complex and robust quantum technologies.

While this study demonstrates a significant advancement, some limitations exist. The experiments focused on a single qubit, and scaling to larger systems poses a challenge. The hardware limitations of the FPGA might restrict the complexity of the neural network and the size of the quantum system that can be controlled effectively. Further investigation is needed to determine the scalability of the approach. The specific reward function used might affect the learned strategies, and exploring alternative reward functions could potentially reveal different optimal control protocols. The fidelity of the initialization was partly limited by the rethermalization of the qubit; more sophisticated strategies might be necessary to fully mitigate this effect in larger systems. Finally, the current setup only handles certain specific control tasks and may need modifications to suit other scenarios.