Conference Agenda

Reinforcement learning (RL) for chemical process control aims to enable more optimal plant-wide decision-making while maintaining safety in hazardous scenarios. However, process settings are both highly complex and more sample-constrained than other established applications of RL, such as fine-tuning language models or silicon chip design, which both apply neural networks (NNs) for decision-making. As opposed to NNs, using Gaussian processes (GPs) to approximate state-action value functions in sample-constrained applications can mitigate against over-fitting and provide analytical uncertainty estimates, enabling probabilistic constraint handling which is critical for safe learning in industrial processes. Inherent uncertainty quantification also enables automatic exploration and exploitation through an acquisition function. As GPs are a non-parametric distribution over potential functions, they reduce overheads in architecture design and hyperparameter tuning, both ill-defined tasks with small datasets.
Previous work has demonstrated the potential of GP models for sample-efficient reinforcement learning in chemical process control and robotics [1,2]. GPs are used to learn the Q-function, which quantifies the expected future reward from taking an action in a given state. However, these previous attempts do not distinguish between inaccurate Q-values generated early on, hindering policy improvement and slow convergence. While the parameters of NNs continually update throughout RL as the distribution of states, actions, and rewards shift, non-parametric GPs must analogously be able to 'forget' inaccurate early representations of the Q-function.
In this work, we enable RL for sample-constrained process control using GPs. We introduce a subset selection mechanism that dynamically selects previous trajectories and reward profiles, balancing coverage while maintaining the density of data high-performing regions of the state-action space, and omitting inaccurate suboptimal assessments of the Q-function. The mechanism we propose is based on the M-Determinantal Point Process (M-DPP), which defines the probability of a subset's selection according to the determinant of its associated Gram matrix [3]. By applying developments from the Sparse Bayesian Optimization community [4], we incorporate Monte Carlo estimates of the state-action values of data points into this selection.
By mitigating against 'inaccurate' initial representations of the Q-function, this work addresses a key limitation of current GP Q-learning methods, where early, suboptimal trajectories could unduly influence the Q-function approximation even after the policy has improved. By selectively 'forgetting' data, our proposed approach allows the GP to more accurately model the Q-function for the current policy, leading to faster convergence and improved final performance. The adaptive subset selection introduced in this work represents a key step toward the prevalence of RL for industrial process control. These enhancements address key challenges in sample efficiency and computational scalability, leading to more optimal plant-wide decision-making while maintaining safety and reliability.
[1]T. Savage, et al. Model-free safe reinforcement learning for chemical processes using gaussian processes. IFAC-PapersOnLine, 2021.
[2]M. P. Deisenroth, et. al.. Gaussian Processes for Data-Efficient Learning in Robotics and Control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015.
[3]D. R. Burt, et. al . Convergence of sparse variational inference in gaussian processes regression. JMLR, 2020.
[4]H. B. Moss, et. al. Inducing point allocation for sparse gaussian processes in high-throughput bayesian optimisation. AISTATS, 2023.

Process monitoring is crucial in industrial facilities to maintain stability, meet production targets, and fulfill safety and environmental regulations. This typically involves regulating variables such as temperatures, pressures, flow rates or levels. However, when composition measurements are required, their evaluation becomes challenging. Composition sensors usually show several drawbacks, including high capital and operating costs, short lifetimes, sensitivity to harsh environments, low data frequency and frequent calibration requirements. To tackle this problem, several alternatives of indirect composition monitoring have been reported in literature. Among these, soft sensors have gained attention in recent years due to their accuracy, ease of training, and the potential of integrating widely known machine learning techniques. In this study, we describe the methodology adopted to train a soft sensor for hydrogen sulfide monitoring in an industrial oil regeneration facility located in Pieve Fissiraga, Italy. The plant currently measures the hydrogen sulfide concentration through sampling and subsequent analysis using a gas chromatograph, which has led to significant delays in the composition measurement of up to eight hours. Unfortunately, insufficient historical data was available to correlate the laboratory analyses with measured plant variables. As an alternative, the data was used to develop and validate a rigorous simulation of the process using Aspen HYSYS. Although the simulation demonstrated high accuracy, with errors of around 2% when compared to plant data, its complexity associated to the Aspen HYSYS acid gas package used led to long computational times and convergence issues. For this reason, we adopted a data-driven surrogate-modeling approach. The surrogate model, based on Kriging Gaussian Process, was trained using data extracted from the simulation in a space-filling design derived from Latin Hypercube Sampling. The model demonstrated a high fidelity to the process simulation, with prediction errors of less than 3%, providing a practical and cost-effective soft-sensor for real-time hydrogen sulfide monitoring with the potential to significantly reduce off-spec operation times.

The key innovation in this research lies in the combination of process simulation, surrogate modeling, and data pre-processing to develop an accurate soft sensor for hydrogen sulfide monitoring. The historical plant data was pre-treated with an innovative approach to tackle its noisy and unsteady behavior and determine the steady states of the plant. Finally, the developed soft sensor is expected to be validated in the industrial environment, enhancing process control and improving environmental compliance.

References

• S. Chen, C. Yu, Y. Zhu, W. Fan, H. Yu, T. Zhang, 2024. NOx formation model for utility boilers using robust two-step steady-state detection and multimodal residual convolutional auto-encoder. Journal of the Taiwan Institute of Chemical Engineers 155, 105252.
• A. Galeazzi, F. de Fusco, K. Prifti, F. Gallo, L. Biegler, F. Manenti, 2024. Predicting the performance of an industrial furnace using Gaussian process and linear regression: A comparison. Computers & Chemical Engineering 181, 108513.
• Y. Jiang, S. Yin, J. Dong, O. Kaynak, 2021. A Review on Soft Sensors for Monitoring, Control, and Optimization of Industrial Processes. IEEE Sensors Journal 21, 12868–12881.

Process optimisation and quality control are crucial in process industries for minimising product waste and improving overall plant economics. Identifying robust operational regions that ensure both product quality and performance is particularly valued in the chemical and pharmaceutical sectors. However, this task is complicated by uncertainties such as feedstock variability, control disturbances, and model mismatches, which can lead to violations of product quality constraints and significant batch discards. Addressing these uncertainties is essential for maintaining process stability and maximising profitability, as uncontrolled variability introduces stochastic elements into product quality that can result in large-scale wastage if not managed effectively.

We propose a novel robust optimisation strategy that integrates advanced machine learning and process systems engineering techniques to systematically identify optimal operational regions under uncertainty. Our approach begins by using a process model to screen a broad operational space across various uncertainty scenarios, pinpointing promising control trajectories to satisfy process constraints and product quality. Machine learning is then employed to cluster these trajectories into sub-regions. Meanwhile, correlations between key control variables are quantified through interpretable AI methods to reduce the operational space dimensionality. Finally, a two-layer dynamic optimisation framework is employed to determine the optimal control trajectory and its corresponding operable space within each promising sub-region.

To demonstrate the efficiency of our approach, we used a case study focusing on the quality control of a dynamic batch process for formulation product manufacturing. This case studies accounts for generic industrial uncertainties such as feedstock variation, control disturbances, and operator human errors. The results from this case studies highlights the advantages and industrial potential of our proposed strategies, indicating their significant promise for industrial application.

Heat transfer enhancement techniques are frequently used in heat recovery systems with shell and tube heat exchangers. One of the most common methods is the incorporation of turbulence promoters, which increase thermal efficiency but also cause flow disturbances that result in an increase in pressure drop [1]. The design of shell and tube heat exchangers with turbulence promoters, such as perforated twisted tapes [2], is determined by several geometric variables, such as the inner and outer diameter of the tube, its length, the shell diameter, the baffle spacing, the baffle cut percentage, and the promoter geometry. A key objective in the design of heat exchangers is to minimize their total cost. The thermohydraulic performance of the exchanger depends directly on the geometry of the shell, tubes, and turbulence promoter, making it crucial to determine the optimal combination of parameters to calculate flow rates, the overall heat transfer coefficient (U), and pressure drops [3]. In this work, a deterministic approach is proposed to optimize the design of the heat exchange device. The objective function is to minimize the total annualized cost (TAC) of the equipment. The model is solved in the software GAMS, employing the solver CONOPT. Three case studies are presented: 1.- Optimization of a water-water heat exchanger with similar mass flow rates, 2.- Optimization of a water-water heat exchanger with different flow rates, and 3.- Optimization of a heat exchanger for water-waxy residue. In the first case, it was observed that in the system with turbulence promoters, the overall heat transfer coefficient (U) increased by 53% and the transfer area decreased by 34%, leading to a 64.23% increase in pressure drop inside the tubes. For the water-water system with different mass flows, the costs of an optimized smooth tube system and an optimized system with turbulence promoters were similar, regardless of the fluid arrangement, with an average TAC of 1,384 USD/year. In the case of the waxy residue, the TACs resulted in 12,100 USD/year for the smooth tube and 12,709 USD/year for the tube with the promoter, concluding that the integration of the turbulence promoter did not reduce costs due to a 21% increase in pumping costs.

REFERENCES

[1] M. Picón-Núñez, J. I. Minchaca-Mojica, and L. P. Durán-Plazas, Selection of turbulence promoters for retrofit applications through thermohydraulic performance mapping, Thermal Science and Engineering Progress, vol. 42, Jul. 2023, doi: 10.1016/j.tsep.2023.101876.

[2] M. M. K. Bhuiya, M. S. U. Chowdhury, M. Saha, and M. T. Islam, Heat transfer and friction factor characteristics in turbulent flow through a tube fitted with perforated twisted tape inserts, International Communications in Heat and Mass Transfer, vol. 46, pp. 49–57, Aug. 2013, doi: 10.1016/j.icheatmasstransfer.2013.05.012.

[3] R.K. Sinnott, Chemical Engineering Design, Coulson & Richardson’s Chemical Engineering Series, Volume 6, 4^th edition, Elsevier, 2005.

Abstract

The design of structured packings for thermal separation columns has been the subject of extensive research for 60 years (Spiegel & Meier, 2003). Despite the profound expertise and considerable advances that have already been gained in this field, it remains a challenging but promising task. This is due to the fact that the packing performance depends on a variety of fluid dynamic and mass transfer-related parameters, which exert a mutual and not necessarily beneficial influence on one another. The systematic development of improved packings by means of computational fluid dynamics (CFD) simulations is a promising approach to yield improved designs that can be manufactured on the basis of the derived CAD models.

Various research groups conducted parameter variations based on CFD simulations to systematically design structured packings (Neukäufer et al., 2019; Zawadzki et al., 2023). However, only a limited number of studies have employed mathematical optimization for this purpose. In previous work, two different CFD-based optimization approaches were developed with the objective of minimizing the pressure drop while maintaining or even maximizing the specific surface area, which is considered as indicator for mass transfer performance (Lange & Fieg, 2022). In topology optimization the material distribution is varied in a predefined grid structure making use of an evolutionary algorithm. While this approach provides high potential for innovation it can only produce rough design drafts for tractable grid sizes. In contrast, shape optimization modifies a given structure gradually based on gradient information of the objective function, as obtained through adjoint simulations (Othmer, 2008). This approach does not enable the generation of new structures, but can be beneficial as a refinement tool for further improvement of already well-performing packings.

In this contribution, shape optimization coupled with single-phase CFD simulations of the gas phase is used to improve two initial packing structures. By starting from a topology-optimized packing structure the sequential integration of both optimization methods is evaluated, while the transferability of the shape optimization approach to established structured packings is evaluated on the basis of an initial Rombopak packing. For both cases, a successful application of the shape optimization was realized, resulting in reshaped packings with constant surface area, reducing the pressure drop by 16% for the topology-optimized packing and 3% for the Rombopak. The analysis of the shape-optimized packings provides further insight to the specific modifications that resulted in these improvements, which includes rounding of edges and the closure of dead zones.

References

L. Spiegel, W. Meier, 2003, Chem. Eng. Res. Des., 81, 1, 39-47.

J. Neukäufer, F. Hanusch, M. Kutscherauer, S. Rehfeldt, H. Klein, T. Grützner, 2019, Chem. Eng. Technol., 42, 9, 1970-1977.

D. Zawadzki, M. Blatkiewicz, M. Jaskulski, M. Piątkowski, J. Koop, R. Loll, A. Górak, 2023, Chem. Eng. Res. Des., 195, 508-525.

A. Lange, G. Fieg, Novel Additively Manufacturable Structured Packings Developed by Innovative Design Methods, The 12^th international conference Distillation & Absorption, Toulouse, 18.-21. September 2022.

C. Othmer, 2008, Int. J. Numer. Meth. Fluids, 58, 8, 861-877.