A Probabilistic Reinforcement Learning Framework for Optimized Decision-Making in Geosteering

Abstract

Geosteering refers to the process of intentionally adjusting the drilling trajectory to navigate subsurface environment and maintain alignment with target formations. It is a key process used while drilling oil and gas wells but is also gaining traction in other areas, such as geothermal and civil tunnels drilling. Geosteering is fundamentally a sequential decision-making process, where a series of steering decisions are made under uncertain conditions to optimize the drilling trajectory in real-time. By framing it as a sequential process, operators or decision-makers gain the flexibility to make their decisions as new data becomes available and uncertainties resolved during the drilling operation. In the oil and gas industry, the utilization of real-time logging-while-drilling (LWD) data has significantly enhanced the ability to make informed decisions during the geosteering process. Most recent research efforts focus on automating and improving the accuracy of LWD data interpretation to better estimate subsurface conditions. However, the challenge remains to effectively use these estimates in a way that optimizes steering decisions and enhances overall operational efficiency. This dissertation tackles this challenge by developing an automated decision-making framework for geosteering. The framework uses reinforcement learning (RL), a subset of machine learning, to optimize and automate the decision-making process. The first contribution of the dissertation is validating the suitability of RL for geosteering decision-making by performing a comparison study against existing methods. The study shows that our RL-based geosteering framework, particularly with the deep Q-network (DQN) algorithm, consistently outperforms greedy optimization, which focuses on short-term gains. Furthermore, the framework provides results comparable to approximate dynamic programming (ADP), but with significantly reduced computational demands, especially after the training phase is complete. We also introduce the RL-Sensor method, which optimizes geosteering decision-making by utilizing data from behind the decision points, thereby eliminating the need for the Bayesian framework for estimating data ahead of the decision points. This significantly reduces computational demands, particularly during the training phase. The second contribution of this dissertation focuses on extending the RL-based geosteering framework and applying it to realistic, field-scale scenarios. This includes the development of the RL-Estimation method, which integrates the particle filter (PF), a state estimation method, into the framework. By combining real-time state estimation with probabilistic estimates, the RL-Estimation method enhances decision-making under uncertainty, significantly improving the robustness and reliability of the decision-making process. Additionally, the dissertation introduces the “Pluralistic” geosteering robot, which applies the extended RL-based geosteering framework to realistic geosteering contexts. This robot adapts the framework to industry-standard geosteering software, incorporating targetline action spaces and realistic DLS constraints. Trained on stochastic geological models informed by human experts interpretations, the robot has demonstrated performance that exceeds that of top-quartile human experts in synthetic test environments. In summary, this dissertation bridges the gap between the theoretical potential of RL and its practical application in real-time geosteering decision-making. It provides a solid foundation for future advancements in the RL-based geosteering framework, contributing to more efficient and reliable automated geosteering decision-making in the oil and gas industry, as well as other drilling sectors.