Artificial Intelligence-based Module Type Package-compatible Smart Sensors in the Process Industry Laura M. Neuendorf*, Valentin Khaydarov, Christiane Schlander, Tobias Kock, Joshua Fischer, Leon Urbas, and Norbert Kockmann DOI: 10.1002/cite.202300047 This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. Supporting Information available online Image analysis presents a set of powerful methods to receive additional information about multiphase processes. It enables the development of advanced applications for process monitoring and optimization or, so-called, soft sensors. However, the integration of advanced smart sensor systems based on image analysis into the process control system presents a com- plex task. To address this challenge, a modular automation concept offers a standardized interface to integrate modules. This paper presents an integration profile as a service specification that allows a plug-and-measure integration of smart visual sensors into modular plants. To verify the concept, we applied it to three different use cases. At the end, we discuss open challenges in the integration of complex analysis systems with multidimensional data streams into modular plants. Keywords: Artificial intelligence, Complex data streams, Computer vision, Image processing, Modular automation, Module type package Received: March 10, 2023; revised: June 28, 2023; accepted: July 20, 2023 1 Introduction Highly volatile market dynamics, long-term uncertainties in supply chains and increased demand for individualized products are presenting critical challenges in the chemical and, especially, pharmaceutical, and special chemistry industries [1]. Modular automation and the module type package (MTP) are key technologies to address them by ensuring high flexibility of the production facilities [2]. Pre- designed self-contained process equipment assemblies (PEA) are connected to a modular plant and integrated into the process control system or, so-called, process orchestra- tion layer (POL). At any time, PEAs can be rearranged and used in another production line. Thus, great flexibility is provided. This paradigm requires an access to a high variety of modules that are ready to integration. So far research and development in the field of modular automation was mainly focused on process-related modules such as dosing units, stirred tank units, or tempering modules. Although recent advances in image analysis empowered by machine learning provide new opportunities to use data to generate models for advanced process monitoring [3, 4]; there are only single developments in the integration of smart sensor systems into modular plants [5]. In this paper, we present a concept of a smart image anal- ysis module for advanced process monitoring. In the first section of the paper, we provide a literature review on mod- ular automation as well as artificial intelligence (AI) focus- ing on computer vision applications in the chemical indus- try. The second section presents the developed service concept for the camera module with a detailed description of services, procedures, and parameters. Then, we present three reference use cases that are used to evaluate our con- cept. The last section presents a conclusion and discussion on new challenges to the MTP interface derived from the considered use cases. www.cit-journal.com ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 – 1Laura M. Neuendorf https://orcid.org/0000-0001-6593-1752 (laura.neuendorf@tu-dortmund.de), 2Dr. rer. nat. Valentin Khaydarov, 3Christiane Schlander, 2Tobias Kock, 3Joshua Fischer, 2Prof. Dr.-Ing. Leon Urbas, 1Prof. Dr.-Ing. Norbert Kockmann 1TU Dortmund University, Department of Biochemical and Chemical engineering, Laboratory of Equipment Design, Emil-Figge-Straße 68, 44227 Dortmund, Germany. 2TU Dresden University, Process-to-Order Lab, Helmholtzstraße 16, 01069 Dresden, Germany. 3Merck Electronics KGaA EL-OTE, Frankfurter Straße 250, 64293 Darmstadt, Germany. 1546 Research Article Chemie Ingenieur Technik http://crossmark.crossref.org/dialog/?doi=10.1002%2Fcite.202300047&domain=pdf&date_stamp=2023-08-09 2 State of the Art In this section, we give a brief review of modular automa- tion and the development of modules. After that, the topic of AI in the chemical industry is considered. 2.1 Modular Automation Modular automation offers promising technologies for the process and chemical industry. It meets the requirements for greater flexibility and a shorter development cycle for production facilities by dividing the process plant into self- contained production units or PEA. A key concept of mod- ular automation is the MTP. It represents a standardized and manufacturer-independent description of the automa- tion interface of PEAs and endows their plug-and-produce capability [6, 7]. The functionality of the PEA is defined in the form of serv- ices [8]. Each service encapsulates a self-contained function with a standardized state-based interface. Variants of the service execution are presented as procedures that together share one state-based control in the service. Services and pro- cedures can be parameterized using configuration and proce- dure parameters. Besides that, the procedures have input, output, and report values. The latter is used to provide the POL sensor readings on the plant level for further data his- toricization. A comprehensive state-of-the-art on modular automation can be found in the book of Tauchnitz et al. [9]. Services and procedures together with their parameters and variables compose a service specification that is the most important component of every PEA since it defines the module’s functionality. The process of designing the service specification is a complex engineering process that connects process and equipment knowledge with the requirements from the side of the end user [10, 11]. The developed service specification and description of the automation interface are supposed to present requirements for developing concrete PEAs of this type. This type-based approach for the engineering of modular plants (proposed in [12]) not only accelerates the PEA engineering process by giving PEA manufacturers the template and the automa- tion interface description, but it also drastically increases flexibility. The module within the plant can be replaced by another module of the same type but from another manu- facturer without significant changes in the orchestration. The type-based approach expects that there is a set of abstract module types that share the same functionality. BioPhorum organization proposed such a unified profile for a stirred tank unit [13]. The profile describes the set of serv- ices with their functionality, procedures, parameters, and variables. Authors of [14] are developing a standardized MTP-profile for water electrolysis modules. Although visual sensors or other advanced sensing systems are a common equipment in chemical industries, no unified service specifi- cation has been developed for them by now. 2.2 Artificial Intelligence There is considerable excitement in the chemical industry about advances in data-driven computational methods and the associated expectations for machine learning (ML) and AI. The increase in available data and computational power have made ML an attractive way to generate understanding and predictions from historical information, and it is mak- ing inroads into the process industry. Typically, ML refers to a process that converts experience or training data into expertise in the form of an algorithm that performs some tasks [15]. A wide variety of tasks such as active learning applications for recognizing steps in chemical batch pro- duction are covered by AI applications in the chemical industry [16]. Also, computer vision for tasks of process monitoring in the chemical industry is on the rise. As the processes benefit from improved supervision thus more precise process con- trol becomes possible [17–19]. Image processing applica- tions range from classification tasks to object localization and instance segmentation tasks, where multiple objects need to be found within an image. For example, in the field of segmentation of grains for digitized grinding tools to measure abrasion and thus deter- mine their remaining life span [20]. Another use case in the chemical industry is the usage of Boston Dynamic’s dog-like robot Spot for reading analog gauges at industrial facilities or for thermal anomaly detection [21]. Moreover, in the field of quality control deep learning is being used for defect detection to improve performance and reduce quality inspection costs by process automation [22]. 3 Service Concept Once an abstract, unified service design has been com- pleted, the PEA hardware can be adapted to specific process conditions by using appropriate equipment. Although the use cases considered further utilize different hardware com- ponents, the service specification and automation interface (MTP) remain the same. This way the requirements of flexi- bility and short integration time of modular plants can be fulfilled. To develop an abstract service concept the required func- tionalities were first evaluated within a series of workshops. ‘‘Service’’ is a concept derived from the VDI guideline VDI 2776 [6] that describes abstracted process functions. The smart sensor should be able to acquire, process and archive relevant data. In addition, hardware-specific func- tionalities are defined which can be utilized if needed and the required hardware is available, e.g., adjustable illumina- tion or a pan and tilt head to move the camera. Ongoing from this, relevant services are specified as shown in Tab. 1. The service Raw-data acquisition is developed to enable the PEA to capture images or obtain other types of input data and prepare them for further processing, e.g., cropping Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH www.cit-journal.com Research Article 1547 Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense www.cit-journal.com ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 Table 1. Description of the developed services and their respective parameters. Parameter Parameter Type Input/ Output Description Service: Raw_data_acquisition Shutter_speed_setpoint AnaServparam In Sets the shutter speed Resolution_setpoint AnaServparam In Sets the captured image resolution ROI_x0 AnaServparam In Variables to cut off the image to important regions ROI_y0 AnaServparam In ROI_x_delta AnaServparam In ROI_y_delta AnaServparam In Gain_setpoint AnaServparam In Sets the gain applied to the captured image Auto_brightness_setpoint AnaServparam In Enables or disables automatic brightness adjusting Time_interval_setpoint If 0 the camera takes images as a sequence as fast as possible In Only relevant for continuous capturing. Defines the frequent of recorded images (zero means continuous Video) Shutter_Speed_feedback AnaView Out Feedback value about the current shutter speed Resolution_feedback AnaView Out Feedback value about the current image resolution Gain_feedback AnaView Out Feedback value about the currently applied gain Auto_Brightness_feedback AnaView Out Feedback about the applied automatic brightness Webserver_endpoint StringView Out Shows the web address of the captured image stream Service: Raw_data_archiving Data_sink StringServParam In Specifies where the captured images are stored Data_format StringServParam In Specifies in which format the images are stored Status message StringView Out Status message to inform, e.g., about low remaining memory Service: Data processing Model_ID DIntServParam In Identifier which model should be used to process the captured images Result AnaView Out Result of the model after processing the last image Confidence_interval AnaView Out Confidence of the model when available Status message StringView Out Status message, e.g., for displaying relevant parameters or proc- essing times Service: Camera Positioning X_setpoint AnaServParam In Input to move the camera it can be either absolute/ relative coor- dinates or absolute/ relative angles, describing the rotation of the camera around all three axesY_setpoint AnaServParam In Z_setpoint AnaServParam In Position ID DIntServParam In Input to move the camera to a previously saved position X_current AnaView Out Current absolute coordinates or angles of the camera Y_current AnaView Out Z_current AnaView Out Service: Configuration Mode Configuration_ID DIntServParam In Input to load a predefined configuration Current_Configuration_ID DIntView Out Displays the currently active configuration ID 1548 Research Article Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense an im- age around the important region. The data acquisition serv- ice covers the data acquisition functionality of the PEA. The service has three procedures to enable the camera’s different operating modes to take continuous/single or trigger-based images. Also, different procedure parameters are implemented to allow configuring the image resolution, region of interest (ROI), brightness, gain, and image captur- ing frequency. As report values the current values of the above-listed parameters are available for reporting pur- poses. Additionally, a web server endpoint is published as a string variable. Thus, the image from the camera in the real-time can be translated over a network for visualization or other purposes. The service Raw-data acquisition presents a core service of the module. Images captured during its execution are available for other services and can also enable their execu- tion. For example, while another service Data processing is running, the internal processing loop will be executed each time a new image is taken within the service Raw-data acquisition. Thus, the image processing routine can be synchronized with image acquisition. Data processing processes the acquired data using a selected image analysis technique or deployed model. The service consists of two procedures that rely on the output data types, regression and classification. The regression model returns a continuous value, while the classification model outputs an enumeration value. As a procedure parameter, the user specifies a model ID that decides which of the available models should be inferred. As described above, each time the service Data acquisition is triggered, the data analysis routine is executed. Adding, modifying or removing of models is done outside the normal operation and therefore not part of this service. Data archiving is designed to save the generated input and output data. The data archiving service is not defined as a main functionality of the PEA. The service enables the PEA to archive the obtained frames in a defined format to a variable data sink. The service depends on the data acquisi- tion service to obtain images. If the service is running and new frames are detected an internal variable is set to true and if the frame is successfully archived, it is set to false. When a new frame is captured by the acquisition service the variable is set to true again. This archiving does not replace the documentation done by the POL because only the pictures and associated parameters are saved and not all service states and parameters. The service is furthermore intended to obtain training images for AI models under conditions as close to the real use case as possible. Data configuration service is intended as a function to set up specific sensor parameters, camera models etc. which are not implemented in the data acquisition service. Addition- ally, when in execute it allows the modification and/or removal of AI models by accessing them with their ID. Illumination enables the PEA to influence the lighting conditions to obtain images as good as possible. The illumi- nation service is dedicated to the illumination functionality of the PEA. It contains three procedures to realize continu- ous, interval-based, or trigger-based illumination. Further- more, it enables the user to define the wavelength and the intensity. It also delivers feedback on the current intensity. This service has to be adjusted in its functions depending on the used illumination hardware and the existing interfa- ces. Camera movement to enable the PEA to move the camera to a different location inside the laboratory setup. It has three procedures, the first one is capable to move the device to absolute position coordinates by taking them as proce- dure parameters. The second procedure uses relative posi- tion inputs to change the position-based on the actual one. The third procedure deals with more advanced position Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH www.cit-journal.com Parameter Parameter Type Input/ Output Description Service: Illumination Wavelength_setpoint AnaServParam In Sets the emitted light color as wavelength Intensity_setpoint AnaServParam In Sets the light intensity Frequency_setpoint AnaServParam In Sets the blinking frequency Duration_setpoint AnaServParam In Sets the time on cycle time Intensity_feedback AnaView Out Feedback about the current light intensity Light_trigger BinProcessValueIn In Used to switch the light on and off by an external source Service: Lens Focus_setpoint AnaServParam In Defines the focus point or sets it to autofocus Iris_setpoint AnaServParam In Define the iris set point or sets it to auto Focus_feedback AnaView Out Feedback about the current focus Iris_feedback AnaView Out Feedback about the current iris Table 1. Continued. Research Article 1549 Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense changes, to find the desired position either by using saved coordinates or by orienting on markers on the plant, e.g., QR codes. Lens control to define optical parameters like focus or aperture. As the name suggests the camera lens service should manage the parameters of the optical lens in front of the camera. There are two modes depending on whether the camera model and lens can be accessed via a digital interface or have to be set up manually. The service only has one procedure which sets the relevant parameters, such as the (auto) focus set point. Once an abstract, unified service design has been devel- oped, the PEA hardware can be adapted to specific process conditions by using appropriate equipment. The here con- sidered use cases utilize different hardware components, but the automation interface (MTP) remains the same for all PEAs of this PEA type. This way the requirements of flexi- bility and short integration time of modular plants can be fulfilled. The core smart sensor PEA functions are covered by the data acquisition, the data archiving, and the data processing service. Starting from them the function of the smart sensor PEA can be enhanced with further services when needed. 4 Reference Use Cases The developed service concept is now applied to three inde- pendent use cases in which the MTP-capable soft sensor enhances different existing fully automated plants. 4.1 Process Development Supported by Monitoring (Merck) In industry, modular development allows highly flexible plants. On the automation layer, this modularity could be realized using MTP and POL. To achieve the goal of the proposed concept, Merck developed a PEA which can be integrated into the existing POL Zenon Logic from Copa Data. The main benefits of this module are monitoring, image acquisi- tion, and user-specific applica- tions in chemical fume hoods. Therefore, the PEA consists of a smart camera Iris GTR5000c from Matrox, a TURCK pro- grammable logic controller (PLC), and additional elements for adjusting position, focus, and lighting. The structure of the communication between the dif- ferent elements is illustrated in Fig. 1. The state logic of MTP is loaded into the PLC, which com- municates with the POL via MTP protocol. The MTP inter- face is created according to the VDE/VDI standards with the open-source software CodeSys and the installed library MTPLib from SiSa. Modbus/TCP is used to communicate between the PLC and the smart camera. The smart camera is able to perform AI classification or image analysis meth- ods, depending on the current application. The file-sharing system Samba and additional access rights allow images to be saved on the POL server, with the currently stored images assigned to the MTP log. For live monitoring, the smart camera provides a web server from which the POL retrieves the current image and displays it on the POL’s human machine interface (HMI). A sketch of the HMI is shown in Fig. 1. The attached elements liquid lens EL-16-40-TC-VIS-5D-C from Optotune and light are con- trolled via serial interface RS232 and via Input/Output (IO). Between the liquid lens and PLC there is an additional lens controller TR-CL 180 from Gardasoft. Along with the elec- trically controllable elements, other manually controllable elements such as an objective lens and an adjustable arm for positioning the camera are also attached. The PEA consists of three services RawDataAcquisition, RawDataArchiving, and DataProcessing. RawDataAcquisi- tion includes two procedures FreeRun and SnapShot. FreeRun starts the live monitoring of the smart camera, SnapShot acquires one image and shows it on the HMI of the POL. In addition, this service provides settings for expo- sure time, focus, and illumination via procedure parame- ters. Since the modules receive an IP address via DHCP, the configuration parameter CamIP can be set to establish the Modbus connection between PLC and smart camera. The service RawDataArchiving has two procedures SingleSave and IntervalSave. SingleSave stores one actual image, IntervalSave stores multiple images with time-adjustable intervals. A procedure parameter allows the user to specify the path where the images are saved on the POL server. The DataProcessing service controls the currently used method for image analysis or AI classification. The methods run on the smart camera in the Matrox Design Assistant (MDA) www.cit-journal.com ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 Figure 1. Left: Camera module setup at Merck. Right: Sketch of the POL HMI. The camera icon symbolizes the button for opening the live monitoring. 1550 Research Article Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense software. MDA has already integrated image analysis func- tions and supports trained AI classifications, which can be used for several applications. 4.2 Hydrodynamics Supervision in a Solvent Extraction Process (TU Dortmund University) Continuous process supervision is of great interest for nearly any (bio-)chemical process. In our laboratory, an artificial intelligence-based optical sensor was developed to monitor a solvent extraction process. A DN32 extraction column, depicted in Fig. 2, was used [17, 23, 24]. By using image-based sensors more insights into the hydrodynamics of the extraction process are obtained [27]. By detecting droplets and calculating their average size, supervision of whether the process is running in the ideal hydrodynamic state is performed. Even overlapping drop- lets can be detected by the neural net. An image showing one example of the process is evaluated in 0.2 s per image. Therefore, Nvidia RTX5000 GPU on an Intel� Xeon� W-2155 CPU with 10 cores at 3.30 GHz is used. By improv- ing the knowledge of the ongoing process further online process control can be applied. The droplet detection AI algorithm was implemented into a PEA as follows. The setup of the PEA consists of two services, Data acquisition and Data processing, with one procedure each (see Fig. 3). The service Data acquisition with its procedure Free run realizes the PEA functionality to acquire a data stream of images from the connected camera. The procedure param- eter data source allows for selecting the camera or alterna- tively a folder with images. The procedure parameter Region of Interest enables cropping of the original image by input- ting the required coordinates, Image Crop Origin X/Y Direction, and the image cropped width and cropped height coordinates. The service writes the corresponding cropped images to an internal variable as a NumPy array. In addi- tion, a binary flag is set to indicate the generation of a new image. The service Data processing with its procedure drop size detec- tion is dedicated to analyzing the image using the neural net Mask- RCNN [28]. The AI algorithm takes the new images and outputs the average drop size and the bounding boxes of all the drop- lets found [27]. The masked over- laid image is written on the web server and is thus available for the operator. The average drop size is written to the report value average drop size. Ongoing from the defined service architecture the realiza- tion of the PEA with the already developed AI algorithm took around four to six hours of work. For the implementation of the PEA dedicated logic defined in VDI/VDE-2658 the open-source Python package MTPPy was used. [29] Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH www.cit-journal.com Figure 2. Image analysis in a stirred solvent extraction column. Left: Visualization of detected droplets by the AI algorithm. Right: DN32 solvent extraction column setup with camera setup [17, 25, 26]. Figure 3. Extraction drop size detection PEA. Research Article 1551 Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense 4.3 Flow Regime Classification in Bioreactor (TU Dresden University) Monitoring of flow regimes in aerated stirred tanks is an important task to ensure optimal aeration conditions. Low impeller speed results in a lack of energy for sufficient bubble dispersion or so-called flooding. On the other hand, if the agita- tor speed is over-controlled and the completely dispersed flow regime occurs, excessive energy consumption would lead to unfavorable opera- tional costs. Therefore, it is advisable to operate a reactor within the loaded flow regime, that presents a good balance between aeration perfor- mance and energy consumption [30]. One of the promising non-invasive online approaches to estimate the flow regime is the use of image data and a deep learning model as proposed by Kröger et al. in [19] The approach is schematically presented in Fig. 4. During the model development and deployment follow- ing challenges occurred: – The position of the camera impacts the model perfor- mance. Each time the camera sensor is used, its position should be the same as during the acquisition of training data to achieve the best model performance. The use of data augmentation can help to make the trained model more invariant regarding small camera position changes. – The camera parameters such as shutter speed, gain, and gamma define the quality of data. Those parameters need to be adjustable by the operator to ensure high data quality. – The collection of data for training should be synchro- nized with the plant operation cycle to collect only data that are relevant for model training. The above-mentioned challenges were addressed by applying the presented approach of a smart camera with corresponding services and equipment. The final setup of the camera module is depicted in Fig. 5. We used a smart camera VAX-50C.I.NVX (Baumer) with an integrated NVidia Jetson Xavier board (6-core Nvidia Carmel ARM with 384 CUDA cores and 8 GB RAM), where software for camera and position control, data collection, model serving, and MTP interface run. The camera is mounted in a robot arm xArm 5 (ufactory) to ensure re- peatable camera positioning. The service Camera position provides this functionality using an Aruco marker for the precise alignment of the camera to be orthogonal to the plane of the reactor’s win- dow. Then, given a desired angle on the horizontal plane and a distance between the camera and the window, the robot arm moves the camera to the given position. The position is then also provided as additional metadata to the taken image. This allows to consider the camera position in data analysis. To set the camera parameters, the service Data acquisi- tion has corresponding configuration and procedure pa- rameters. Available Python API (NeoAPI by Baumer) for the smart camera allows to programmatically change the www.cit-journal.com ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 Figure 4. Flow regime classification using image data and convolutional neural network. Figure 5. Camera module setup at TU Dresden: the smart camera is mounted in the robot arm to enable precise positioning and repeatability. 1552 Research Article Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense camera parameters. The same API is also used to take images and provide them as NumPy arrays for model infer- ence within the Data processing service or/and data collec- tion within Data archiving. The services Illumination and Objective/Lens control are not implemented. The lightning conditions stays constant thanks to the use of non-transparent shielding fabric. The used lens has been set manually once and requires no changes. Since MTP standard is natively not compatible with two-dimensional arrays, we used image streaming over HTTP to have a web-based image visualization for the oper- ator. The same stream can be integrated into a POL using the iFrame technology. Deployment of the camera as MTP- compatible module significantly reduced the overall effort of the model development and module usage. Moreover, the camera module is now not bound to a certain applica- tion but can be repurposed to work with another reactor after the model replacement. 5 Conclusion and Outlook In this paper, we presented the unified service specification with loosely coupled seven services. This specification pro- vides a comprehensive scope of the possible functionality of a smart camera sensor for a wide range of applications. To verify that we considered three significantly different use cases, where the presented service concept was imple- mented. The concept was also proven to be equipment and solution independent. Development and evaluation of the use cases indicated a drawback in the MTP information model. While most of the sensors and actuators installed in a plant provide and consume scalar values, image data are two- or three-dimen- sional arrays depending on the number of color channels. Such multidimensional data format is not compatible with the current version of MTP and, therefore, additional chan- nels for archiving or visualization purposes were used as an alternative solution. To allow a native integration of image data into POL, corresponding data assembly types need to be introduced into the MTP standard. Supporting Information Supporting Information for this article can be found under DOI: https://doi.org/10.1002/cite.202300047. Acknowledgment We are grateful for the financial support of BMWK (Ger- man Federal Ministry of Economic Affairs and Climate Action, Support codes 01MK20014O, 01MK20014S, and 01MK20014T). Open access funding enabled and organized by Projekt DEAL. Abbreviations AI Artificial intelligence API Active programming interface DHCP Dynamic host configuration protocol HMI Human machine interface I/O Input/Output IP Internet protocol MDA Matrox design assistant MTP Module type package PEA Process equipment assembly PLC Programmable logic controller POL Process orchestration layer References [1] Modulbasierte Produktion in der Prozessindustrie - Auswirkungen auf die Automation im Umfeld von Industrie 4.0: Empfehlungen des AK Modulare Automation zur NE148 der Namur, ZVEI – Zentralverband Elektrotechnik und Elektronikindustrie e. V., Frankfurt/Main 2015. [2] J. Bernshausen, A. Haller, H. Bloch, M. Hoernicke, S. Hensel, A. Menschner, A. Stutz, M. Maurmaier, T. Holm, C. Schäfer, L. Urbas, U. Christmann, C. Fleischer-Trebes, F. Stenger, atp 2019, 61 (1–2), 56–69. DOI: https://doi.org/10.17560/ atp.v61i1-2.2400 [3] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Comput. Intell. Neurosci. 2018, 2018, 7068349. DOI: https:// doi.org/10.1155/2018/7068349 [4] M. Gärtler, V. Khaydarov, B. Klöpper, L. Urbas, Chem. Ing. Tech. 2021, 93 (12), 2063–2080. DOI: https://doi.org/10.1002/ cite.202100134 [5] V. Khaydarov, S. Heinze, M. Graube, A. Knupfer, M. Knespel, S. Merkelbach, L. Urbas, in 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, Piscataway, NJ 2020. [6] VDI 2776 Blatt 1:2020-11, Verfahrenstechnische Anlagen – Modulare Anlagen - Grundlagen und Planung, Beuth Verlag, Berlin 2020. [7] VDI/VDE/NAMUR 2658 Blatt 1:2019-10, Automatisierungs- technisches Engineering modularer Anlagen in der Prozess- industrie, Beuth Verlag, Berlin 2019. [8] VDI/VDE/NAMUR 2658 Blatt 4:2022-10, Automatisierungs- technisches Engineering modularer Anlagen in der Prozess- industrie, Beuth Verlag, Berlin 2020. [9] MTP Automation of Modular Plants (Ed: T. Tauchnitz), 1st ed., Vulkan-Verlag, Essen 2022. [10] H. Bloch, S. Hensel, M. Hoernicke, K. Stark, A. Menschner, A. Fay, L. Urbas, T. Knohl, J. Bernshausen, Procedia CIRP 2018, 72, 1088–1093. DOI: https://doi.org/10.1016/j.procir.2018.03.037 [11] A. Homay, M. de Sousa, A. Zoitl, M. Wollschlaeger, in 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, Piscataway, NJ 2020. [12] J. Lorenz, A. Klose, I. Viedt, J. Mädler, V. Khaydarov, L. Urbas, in Automation 2022, VDI Verlag, Dusseldorf 2022. [13] R. Polley et al., Automated Facility Plug-And-Play - Stirred Tank Unit Interface Specification, Biohorum, Sheffield 2021. www.biophorum.com/download/automated-facility-stirred-tank- unit-interface-specification (Accessed on February 27, 2023) [14] L. Bittorf, L. Beisswenger, D. Erdmann, J. Lorenz, A. Klose, H. Lange, L. Urbas, A. Markaj, A. Fay, in 2022 IEEE 27th Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH www.cit-journal.com Research Article 1553 Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, Piscataway, NJ 2022. [15] L. H. Chiang, B. Braun, Z. Wang, I. Castillo, AIChE J. 2022, 68 (6), e17644. DOI: https://doi.org/10.1002/aic.17644 [16] A. Ahmad, C. Song, R. Tan, M. Gartler, B. Klopper, in 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, Piscataway, NJ 2022. [17] L. M. Neuendorf, F. Z. Baygi, P. Kolloch, N. Kockmann, ACS Eng. Au 2022, 2 (4), 369–377. DOI: https://doi.org/10.1021/ acsengineeringau.2c00014 [18] N. Blum, V. Krespach, G. Zapp, C. Oehse, S. Rehfeldt, H. Klein, Chem. Ing. Tech. 2021, 93 (12), 1937–1948. DOI: https://doi.org/ 10.1002/cite.202100094 [19] C. Kröger, V. Khaydarov, L. Urbas, in 32nd European Symposium on Computer Aided Process Engineering (Eds: L. Montastruc, S. Negny), Computer Aided Chemical Engineering Series, Vol. 51, Elsevier, Amsterdam 2022. [20] P. Wiederkehr, F. Finkeldey, T. Merhofe, CIRP Ann. 2021, 70 (1), 297–300. DOI: https://doi.org/10.1016/j.cirp.2021.04.051 [21] S. C. Nandakumar, S. Harper, D. Mitchell, J. Blanche, T. Lim, I. Yamamoto, D. Flynn, Bio-inspired Multi-robot Autonomy, arXiv:2203.07718, 2022. DOI: https://doi.org/10.48550/ arXiv.2203.07718 arXiv:2203.07718 [22] J. Villalba-Diez, D. Schmidt, R. Gevers, J. Ordieres-Meré, M. Buchwitz, W. Wellbrock, Sensors 2019, 19 (18), 3987. DOI: https://doi.org/10.3390/s19183987 [23] A. Holbach, E. Çalıs̨kan, H.-S. Lee, N. Kockmann, Chem. Eng. Process. 2014, 80, 21–28. DOI: https://doi.org/10.1016/ j.cep.2014.03.013 [24] L. Hohmann, S. K. Kurt, S. Soboll, N. Kockmann, J. Flow Chem. 2016, 6 (3), 181–190. DOI: https://doi.org/10.1556/ 1846.2016.00024 [25] S. Soboll, I. Hagemann, N. Kockmann, Chem. Ing. Tech 2017, 89 (12), 1611–1618. DOI: https://doi.org/10.1002/cite.201700031 [26] S. Soboll, N. Kockmann, Chem. Eng. Technol. 2018, 41 (9), 1847– 1856. DOI: https://doi.org/10.1002/ceat.201800283 [27] L. Neuendorf, P. Müller, C. Bergeest, N. Kockmann, A. Meijer, C. Schlander, Künstliche Intelligenz (KI)-basierte optische Sensorik für flüssig-flüssig Systeme, in 16. Dresdner Sensor- Symposium 2022, AMA Service GmbH, Wunstorf 2022. DOI: https://doi.org/10.5162/16dss2022/P50 [28] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask R-CNN, arXiv.1703.06870, 2017. DOI: https://doi.org/10.48550/ arXiv.1703.06870 [29] V. Khaydarov, L. Neuendorf, T. Kock, N. Kockmann, L. Urbas, in 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, Piscataway, NJ 2022. [30] M. Warmoeskerken, Gas-liquid dispersing characteristics of turbine agitators, Doctoral thesis, TU Delft 1986. www.cit-journal.com ª 2023 The Authors. Chemie Ingenieur Technik published by Wiley-VCH GmbH Chem. Ing. Tech. 2023, 95, No. 10, 1546–1554 1554 Research Article Chemie Ingenieur Technik 15222640, 2023, 10, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1002/cite.202300047 by T echnische U niversitaet D ortm und D ezernat Finanzen und B eschaffung, W iley O nline L ibrary on [14/11/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense << /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (ISO Coated v2 \050ECI\051) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Warning /CompatibilityLevel 1.5 /CompressObjects /Tags /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages false /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 524288 /LockDistillerParams false /MaxSubsetPct 99 /Optimize true /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness false /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Preserve /UCRandBGInfo /Preserve /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages false /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Average /ColorImageResolution 300 /ColorImageDepth 8 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /FlateEncode /AutoFilterColorImages false /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /ColorImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasGrayImages false /CropGrayImages false /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages false /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth 8 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /FlateEncode /AutoFilterGrayImages false /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /GrayImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasMonoImages false /CropMonoImages false /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects true /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False /CreateJDFFile false /Description << /DEU >> /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ << /AsReaderSpreads false /CropImagesToFrames true /ErrorControl /WarnAndContinue /FlattenerIgnoreSpreadOverrides false /IncludeGuidesGrids false /IncludeNonPrinting false /IncludeSlug false /Namespace [ (Adobe) (InDesign) (4.0) ] /OmitPlacedBitmaps false /OmitPlacedEPS false /OmitPlacedPDF false /SimulateOverprint /Legacy >> << /AllowImageBreaks true /AllowTableBreaks true /ExpandPage false /HonorBaseURL true /HonorRolloverEffect false /IgnoreHTMLPageBreaks false /IncludeHeaderFooter false /MarginOffset [ 0 0 0 0 ] /MetadataAuthor () /MetadataKeywords () /MetadataSubject () /MetadataTitle () /MetricPageSize [ 0 0 ] /MetricUnit /inch /MobileCompatible 0 /Namespace [ (Adobe) (GoLive) (8.0) ] /OpenZoomToHTMLFontSize false /PageOrientation /Portrait /RemoveBackground false /ShrinkContent true /TreatColorsAs /MainMonitorColors /UseEmbeddedProfiles false /UseHTMLTitleAsMetadata true >> << /AddBleedMarks false /AddColorBars false /AddCropMarks false /AddPageInfo false /AddRegMarks false /BleedOffset [ 0 0 0 0 ] /ConvertColors /NoConversion /DestinationProfileName (sRGB IEC61966-2.1) /DestinationProfileSelector /NA /Downsample16BitImages true /FlattenerPreset << /PresetSelector /MediumResolution >> /FormElements true /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles true /MarksOffset 6 /MarksWeight 0.250000 /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PageMarksFile /RomanDefault /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /LeaveUntagged /UseDocumentBleed false >> ] >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.252 858.853] >> setpagedevice