Source: Self Made
Indexes to evaluate risk based metrological performance in companies in the energy sector
(*)Aníbal Barrera García; (**)Michael Feitó Cespón; (***)Roberto Cespón Castro
(*)Faculty of Engineering. University of Cienfuegos
Cienfuegos, Cienfuegos Province, Cuba
abarrera@ucf.edu.cu
(**)Faculty of Engineering. University of Cienfuegos
Cienfuegos, Cienfuegos Province, Cuba
mfeito@ucf.edu.cu
(***)Faculty of Mechanical and Industrial Engineering. Central University of Las Villas
Santa Clara, Province of Villa Clara, Cuba
rcespon@ucf.edu.cu
Reception date: 03/12/2021 – Approval date: 03/30/2021
DOI: https://doi.org/10.36995/j.visiondefuturo.2021.26.01.002.en
ABSTRACT
The metrological performance evaluation is a tool that allows feedback in the management of measurements and the monitoring of the results achieved by the organizations that apply it. In many companies this type of evaluation does not show relevant results, mainly due to limitations in the procedures and indicators used to carry it out.
The objective of this work was to develop metrological risk indices for the evaluation of metrological performance in companies in the energy sector. For the development of the research, three organizations belonging to said sector are selected. From the use in the organizations of the risk approach and interviews, document review, brainstorming and mathematical statistical methods, the metrological risk index and the risk index in metrological management are defined for companies in the aforementioned sector. The results achieved showed that it is possible to create this prototype tool, as well as its usefulness to provide feedback on the management of measurements, in addition to guiding the management decision-making process in metrological matters.
KEY WORDS: Metrological Performance; Index; Risk; Measurements; Energy.
INTRODUCTION
The role of measurements is well known in any production or service process, in domestic and foreign trade, in inventory control and in making transcendental decisions for the economic and social life of a country. Hence the importance of ensuring the accuracy of instruments and measurement systems, and above all, that the measurement results, expressed in the units of measurement of the International System of Units, are reliable, safe, accurate and comparable (Hernández and Reyes, 2013).
Oramas (2014) states that metrology has the ideal goal of obtaining 100% accuracy, although in practice this is almost impossible to achieve. However, all the activity of metrological science revolves around that ideal, that goal, studying, establishing and providing procedures, tools and mechanisms that help us to achieve that maximum accuracy.
Llamosa, Milton and Villareal (2011) explain that the role of metrology becomes relevant when the measurement process is vital in some kind of commercial transaction, in military applications, in the health field, in the production of medicines or food; in testing for civil engineering constructions, in diagnostics to discover the cause of some electrical problem, in works aimed at reaching the rational use of energy; in the routine monitoring of electromechanical, mechanical and electronic systems, in the verification of environmental pollutant limits or radiation level values, in the permanent monitoring of the various physical magnitudes involved in production processes, and especially in the performance of quality tests.
It is evident that metrology is present in practically all activities of life, hence its impact, when performed in a reliable, comparable and safe manner, has a direct impact on the economic, political and social development of a country; so much so that in various situations the development of these tends to be measured from the development they have in metrology (Reyes, Alvarez and Hernandez, 2011).
Due to the importance and the role that metrology plays in the business environment, ISO 10012 "Measurement Management System. Requirements for measurement processes and measurement equipment". This standard contains requirements for implementing a measurement management system, based on the continuous improvement cycle and a process approach, which favors its integration into the overall management system of organizations to achieve product quality objectives, as well as to manage the risk of obtaining incorrect measurement results.
Within the measurement management system is the evaluation of metrological performance, defined as a structured, systematic and periodic assessment procedure of qualitative and quantitative estimation, which allows determining the capacity of the metrological process to adequately manage its resources in accordance with the established objectives and goals, which coincides with López et al.
Sotelo, Sosa and Carreón (2020) state that during this type of evaluation, the results of the process should be analyzed in a determined period of time by means of indicators, instruments, audits, among others.
The implementation of the metrological performance evaluation is carried out in accordance with ISO 10012:2003, based on the Deming cycle, which offers a high level of interaction with the rest of the ISO standards corresponding to the different management systems.
In companies that have implemented these systems, each entity applies its own criteria for the systematic identification and evaluation of metrological aspects. The metrological evaluation can provide information to define whether a metrological aspect is significant or not, and to analyze at each stage the fulfillment of the objectives and goals foreseen in the system, finding validity in the UNE 66180 standard.
In relation to the above and under the concept of metrological performance evaluation, a search for information in recognized databases such as Elsevier, Emerald, Scopus, Scielo, among others, is performed, identifying a group of scientific articles related to the subject, being analyzed in order to identify the relevant contributions associated with the design of a performance evaluation model.
Researches such as those of authors Valencia and Restrepo (2016) propose the use of techniques such as Structural Equation Analysis (SEM) comprising Confirmatory Factor Analysis (CFA) through regression analysis. Others such as Osorio, Herrera and Vinasco, (2008) use the Analytical Hierarchical Process (AHP) to establish different criteria to measure process performance. Other authors such as Sarache, Costa and Martinez (2014) propose a methodology to evaluate performance by means of a management indicator.
None of the previous authors mention the evaluation of metrological performance, but rather propose ways to assess performance in a general way in an organization and do not focus on the measurement management system. Among the researches consulted that deal with the evaluation of metrological performance, the following authors stand out: Beltrán, Muñuzuri, Rivas and González (2010) and Beltrán, Muñuzuri, Rivas and Martín (2011).
The existence of a metrological management evaluation model proposed by Beltrán (2006) is detected, which has been developed in several works of application and experimental analysis collected by Beltrán, González and Domingo (2007), until finally its incorporation in the UNE 66180:2008 standard.
The authors Beltrán, Muñuzuri, Rivas and Martín (2014), state that the model proposed by Beltrán (2006) is based on a linear combination of variables (equivalent to the elements that make up the requirements of a measurement management system), based on the assumption that all contribute with equal strength to the overall maturity level of the organization's metrological management.
In 2014, the aforementioned authors proposed a weighting of the variables, obtaining the relative importance of each of these variables and constructs of the same, as well as their influence on the overall result, achieving a model where the variables that form it are no longer equipotential and each one has a weight assigned according to its importance.
In 2015, it was demonstrated that this model does not take into account two important aspects: the continuous improvement cycle proposed by Deming (used by ISO 10012:2003) and, secondly, the level of maturity in the company's metrological management. It is therefore necessary to restructure the model, making it robust for the evaluation of metrological management in companies, guaranteeing the reliability of their measurements.
It is common for companies to identify the metrological risks in order to carry out a study of the causes of possible threats and probable undesired events, damages and consequences that may occur.
In the bibliography consulted, there are no standards or norms referring to metrological risk assessment, but most companies work with the ISO 31000 standard. This standard provides a general framework for risk management, allowing its improvement and integration into the organization's metrological management system.
The risk approach has been successfully applied in different business management systems such as occupational health and safety, environmental management, among others. This trend is not foreign to the energy sector.
Many countries have energy production as an essential link in their strategic objectives for their development, for which they have facilities that allow them to achieve this goal. Within the management systems of this type of industry, the process of controlling measurement equipment is included, and metrological assurance is essential for its correct operation.
The measurement management system of these companies has under its control all types of instrumentation and existing equipment in each of its processes, so that their monitoring and evaluation allows to obtain reliable, safe and accurate measurements for process control and decision making.
In these industries it is evident that metrological evaluations are carried out, being many of them based on the metrological requirements and demands of the legal basis by means of checklists and not by indicators that quantify the metrological aspects of the system and allow an evaluation of the overall performance, to which is added that risk-based thinking is not always used, a relevant aspect in achieving the quality of management systems.
Based on the aforementioned elements, this work aims to develop metrological risk indexes for the evaluation of metrological performance in companies of the energy sector. For the development of the research, three organizations belonging to this sector were selected: an oil refinery, a thermoelectric power plant and an electric energy trading company.
In relation to the methodology, for the development of this work a quantitative research was carried out, which from the use of a group of techniques and/or tools and followed by the corresponding analysis allowed defining the system of indicators from the calculation of the metrological risk index and the risk index in the metrological management. These indexes are distinctive and novel elements that guarantee not only their evaluation, but also their use in the management of measurements in organizations of the energy sector.
The paper is structured as follows: first, the materials and methods are presented, then the results are presented, i.e., the risk-based indices, their field of application, formulation and calculation method for the evaluation of metrological performance. Finally, the conclusions are presented.
DEVELOPMENT
Materials and methods
In the development of the research, the general methodology for the solution Industrial Engineering problems is used, whose stages are shown below.
Stage I: Problem definition and analysis
The purpose of this stage is to maximize the possibility of isolating and defining the problem satisfactorily. Make a detailed list of the characteristics of the problem, including the constraints. It is characterized by the elicitation, investigation, clarification and analysis of the problems identified. The objectives and the real purpose of the study to be carried out must be explained to all those involved, management and workers.
The main techniques to be used for the definition and analysis of the problem are:
Stage II: Analysis and selection of solution alternatives
Devise and develop as many possible solutions as possible, taking into account the constraints. Ideas and suggestions are sought to guide towards the solution. Techniques that may be useful in this stage include:
Stage III: Solution design
In this stage, the proposed solution alternatives must be evaluated, i.e., quantitatively predict the behavior of each of them with respect to each of the criteria considered. The existence of many criteria and their interdependence sometimes forces to ignore several of them during the evaluation, due to the limitations of intangible criteria, which cannot be expressed quantitatively. The alternative solutions must be compared and, based on the criteria defined by the specialists or analysts, the best possible solution must be selected.
Stage IV: Implementation
This stage involves a delineation of the attributes and behavioral characteristics of the selected solution, the main purpose being to communicate the solution to the people involved, such as:
It is possible for these stages to overlap, for example, several solutions may be conceived during the formulation period. The solution should be recorded clearly and in sufficient detail. The most common medium is the oral or written report.
Results
The results of the research are presented below.
Stage I: Problem definition and analysis
One of the organizations where metrology plays a fundamental role are those in the energy sector, due to the existence of processes with high operational risk, as well as an important number of instruments for their control.
The measurement management system in this type of industry, as in others, includes all the processes where measurements are performed; metrological confirmation processes of the measuring instruments and the necessary support processes, which are those relating to the allocation of responsibilities, training, competence and training of personnel, management and allocation of resources, audits, control of non-conformities and continuous improvement, among others.
Many companies in the energy sector lack a systematic evaluation of their metrological performance that, based on a system of objective and synthetic indicators and from a risk approach, allows to provide feedback to management and guide management decision making with a proactive character.
Stage II and III: Analysis, selection and design of the solution
Definition and formulation of risk indexes
As a starting point, the existing deficiencies in corporate metrological management, particularly within the energy sector, are taken as a starting point for the evaluation of the performance of the measurement management system (MMS) and the use in organizations of the risk approach as a principle for evaluating metrological performance. From these, the metrological risk index (IRM) and the risk index in the metrological management (IRGM) of companies in the energy sector are defined as follows:
For the formulation and definition of the indexes, the possible metrological risks of the SGM for companies in the energy sector were identified, taking as the object of study relevant organizations within the sector, such as: a refinery, a thermoelectric power plant and an electric energy trading company.
The Delphi method was used to identify the risks present in the measurement management system in the aforementioned group of companies belonging to the energy sector. Eight experts were selected, including specialists from the organizations mentioned above. It is significant to note that the average number of years of experience in the activity of these experts is 27. For the description of their competence, a combination of self-assessment and competence assessment is used. In the competency profile carried out, there are no experts with a low competency profile; most of them are between medium and high.
The first two rounds of consultations with the experts (r1-r2) are aimed at identifying possible risks due to metrological components present in the measurement management system of the mentioned companies. Then, through two more rounds (r3-r4), the most significant risks for the MMS are selected.
To verify the concordance of the experts, Kendall's coefficient and the corresponding non-parametric hypothesis test (λ2) are used, where it was obtained as a result that in the work team there is concordance with respect to the list of the metrological risks identified for the selected companies of the energy sector.
The table of average ranges and frequencies is analyzed to reduce factors. With a value of Kendall's W equal to 0.713, the twenty-nine (29) risks are defined and are summarized in the following table.
Table No1: Selected metrological risks
Source: Self Made
Definition of the Metrological Risk Index (IRM)
The level of disaggregation of the IRM is given by: system, metrological components and risk, in ascending order to build the concepts of the index on the basis of risk; a key premise for the formulation of the System Risk Index by Metrological Component and by Risk (see figure 1).
Figure No1: Level of disaggregation for the analysis of the measurement management system
Source: Self Made
The components to be evaluated are those contained in the company's measurement management system, these are: Direction, Resources, Operations and Improvement, considered according to the criteria of the ISO 10012: 2003.
Formulation of the metrological risk index (IRM)
To formulate the IRM, we start from the definition of risk and criteria exposed by the authors López (2010) and González (2014) exposed in expression (1).
(1)
where:
R: Dimension of risk.
Po: Probability of occurrence of the risk.
Md: Magnitude of the damage.
D: Risk detection.
Since the probability of risk occurrence (Po) measures the proportion of risk materialization, it is then defined as the statistical probability of the risk occurring.
(2)
where:
Pr: Probability of the risk, given by the deviations with respect to the reference threshold of the indicators associated with the assessed risks.
The magnitude of the damage that may occur if the assessed risk materializes depends equally on the probability of damage occurring and its severity, expressed mathematically in (3).
(3)
where:
Md: Magnitude of damage.
Sd: Damage severity.
Pd: Probability of damage.
As the IRM is not only focused on the risk but also on the damage that may exist if it materializes, the following is defined from the above expressions:
(4)
By means of a probabilistic analysis of the above expression, it is determined that there may be the possibility that the risk occurs and there is no damage at all, or that the risk occurs and there is damage; which indicates that these two probabilities are determined by the intersection of Pr and Pd.
The immediate consequence of the definition of conditional probability according to Walpole, Myers, Myers and Ye (2012) posit:
(5)
where:
Pr: Probability of the risk, given by the deviations with respect to the reference threshold of the indicators associated with the assessed risks.
P(d/r): Probability of the damage occurring given that the risk materializes.
Then; combining (4) and (5) we obtain the expression (6) to evaluate the metrological risk index (MRI) that constitutes the fundamental expression of the proposal.
(6)
where:
IRM: Metrological risk index.
Sd: Severity of the damage.
Pr: Probability of the risk, given by the deviations with respect to the reference threshold of the indicators associated with the evaluated risks.
P(d/r): Conditional probability of damage with respect to risk, given by the probability that when there is risk there is damage.
D: Risk detection.
Risk probability (Pr)
To determine the statistical probability (Pr), the monitoring indicators associated with each risk are identified from normative and empirical references.
To determine the value of Pr, in the case of qualitative indicators, the authors of the research assign the value according to an ordinal scale (see Table 2) proposed by López (2010) in his research.
In the case of qualitative indicators, for example, evidence, documentation, among others, it is only checked whether it is present or not. Therefore, it is considered that when the analyzed MMS component is evaluated, it is assigned a value of 0.5; on the contrary, when it is not evaluated, it is assigned a value of 1, which means that the risk is likely to materialize.
Table No2: Scale to assign Pr values to associated risks in the case of qualitative indicators
Source: Adapted from López (2010)
When dealing with quantitative indicators, for example, equipment measurements, products, among others, where several measurements can be made, the Pr values are the compilation of the measurements where the indicator exceeds the reference limit, expression (7) based on the application of the probability concept taken from Lopez (2010).
(7)
where:
Pr: Probability of the risk, given by the deviations with respect to the reference threshold of the indicators associated with the assessed risks.
Nm (outside the limit): Number of measurements where the magnitude of the risk exceeds the reference criterion.
Ntm (evaluated): Total number of measurements in the evaluated period.
The statistical probability Pr is determined on a continuous quantitative scale, with values between 0 and 1, depending on the path of the variable. The value 
indicates that the probability is maximum, and 
that there have been no deviations in the indicator by which the metrological risk is measured, which could constitute a danger of the occurrence of damage in the period evaluated.
Severity of damage (Sd)
For the severity of damage, the group of experts uses criteria exposed by ISO 31010:2019, Collective of authors (2007) and Gonzalez (2014). From the above analysis it is decided to take six evaluation criteria (Not harmful; Slightly harmful; Moderately harmful; Harmful; Highly harmful and Extremely harmful) which are evaluated using a standardized Likert-type ordinal scale (Table 3).
The maximum evaluation value of the scale is 1 and the minimum is 0, the maximum number of the scale is divided by the total number of evaluation criteria, in order to determine the evaluation range for these criteria, once the damage has materialized, its severity.
Table No3: Evaluation of the severity of the damage
Source: Self Made
Conditional probability (Pd/r)
To determine the conditional probability, that damage exists given that the risk materializes (Pd/r), the experts, taking into account the description of the damage, associate a probability value to it (table 4), following the criteria used in the construction of the Table 3, with the difference that the minimum value of the scale is 0.1, because there is always a minimum probability.
Table No4: Evaluation of the conditional probability (Pd/r)
Source: Self Made
Risk detection (D)
The possibility of detection is performed by means of the established controls. In the absence of statistical data, the working group uses the evaluation criteria according to the scale stated by Gonzalez (2014) in the table below.
Table No5: Criteria to establish the level of risk detection
Source: González, 2014.
Definition of the Risk Index in Metrological Management (IRGM)
Like the previous index, the level of disaggregation of the IRGM is given by: System, Metrological Components and Risk, in ascending form. For its development, the components to be evaluated in the measurement management system are maintained (see figure 1).
Formulation of the risk index in metrological management (IRGM)
To formulate the IRGM, the starting point is the definition of risk set out in expression (1). The probability of occurrence of the risk (Po) for this index is given by the coefficient of metrological management (Cm), since the occurrence of risks depends to a certain extent on the level of metrological management of the company, i.e., it can materialize due to management problems. It is then defined as the coefficient of metrological management, giving the measure that the risk can occur due to the inability in its management.
(8)
where:
Cm: Risk management coefficient, given by the level of attention of the risk.
Then; combining (6) and (8) the expression (9) to evaluate the risk index in the metrological management (IRGM) that constitutes the fundamental expression of the proposal of this index.
The IRGM is expressed mathematically as:
(9)
where:
Sd: Damage of severity.
Cm: Risk management coefficient, given by the level of attention to risk.
: Conditional probability of the damage with respect to the metrological management coefficient, given by the occurrence of damage due to the inability in management.
D: Risk detection
Metrological management coefficient (Cm)
The management coefficient (Cm) of each risk is obtained by considering the level of attention to the causes associated with the risks. The coefficient is formulated by applying the discrete multicriteria method, based on an additive-multiplicative model, according to the expression (10) used by Lopez (2010).
(10)
where:
Cm: Risk management coefficient, given by the level of attention to risk.
C(i): Specific weighting of each cause according to its importance with respect to the associated risk.
V(c): Assessment of the level of attention to the causes associated with the metrological risks evaluated.
A measurement level is associated to each cause, with the objective of assessing the level of attention in its management. For this, the group of experts takes into account the following criteria:
It is proposed to use an ordinal Likert-type measurement level (table 6) where the normalized value is obtained by dividing each number on the scale by the maximum evaluation value, which, based on the evaluation criteria assigned according to its level of attention to its management, is given a numerical value.
To evaluate the level of attention of the causes the group of experts uses criteria exposed by ISO 31010:2019; López (2010) and González (2014). From the above analysis it is decided to take six evaluation criteria (No attention required; Minimum attention; Low attention; Medium attention; High attention; and Very high attention) that are evaluated using a standardized Likert-type ordinal scale (Table 6). The maximum evaluation value of the scale is 1 and the minimum is 0, the maximum number of the scale is divided by the total number of evaluation criteria, in order to determine the evaluation range for these criteria.
Table No6: Assessment scale for the level of care of the causes
Source: Self Made
The causes are identified using Ishikawa diagrams. To determine the C (i) weightings or weights, it is necessary to rank each of the causes associated with the corresponding metrological risk. To obtain the weights, the Saaty method proposes to perform a first level hierarchy: causes by metrological risk, using the Analytical Hierarchy Process (AHP) by means of the SuperDecisions software.
As the severity of the damage (Sd), the detection level (D) and the conditional probability of 
are values determined by expert perception from standardized measurement tables (table 3, 4 and 5), these variables are determined in the same way as in the previous index.
Risk index by metrological component (IRAM) and system (IRS)
Since the metrological risk index (IRM) and the risk index in metrological management (IRGM) have the same levels of disaggregation, to determine the IRAM and IRS, the system disaggregation of Figure 1 is used, composed of the measurement management system and the metrological components present in each of these processes, as well as the risks associated with them.
Then a first level hierarchization is performed using the Analytical Hierarchy Process (AHP), in order to determine the weights or weightings (w) associated with each of the metrological components of the system. These weights are obtained using the SuperDecisions support software (see Table 7).
Table No7: Weights associated with each of the metrological components associated with the Measurement Management System
Source: Self Made
With these weights, the weighted sum is made for the disaggregated level in Figure 1, taking as a basis the MRI and GMRI previously determined for each metrological risk. Obtaining then the values of the Metrological Component Risk Index (MCRI) and System Risk Index (SRI) from two different approaches (risk materialization and metrological management) (expression 11, 12, 13, 14).
(11)
(12)
donde:
: Risk index per metrological component based on the IRM.
: Risk index per metrological component based on the IRGM.
Wa: Weight of the metrological component.
n: Number of risks per metrological component.
(13)
(14)
where:
: System risk index based on IRM.
: System risk index based on the IRGM.
n: Total MMS metrological components.
Evaluation of the results
For both quantitative and qualitative assessment of the IRM and IRGM results for each risk, metrological component and system, a scale is constructed, where the number of intervals considered is determined by applying the equation proposed by Sturges (15), for a number of assessed risks of N=29.
(15)
where:
N: Sample size
K: Number of intervals
Once the number of intervals has been obtained, the range (R) is determined according to expression (16) and the amplitude (C) according to (17)
(16)
where:
R: Range
Value (max.): Maximum value of the IRM, IRGM.
Value (min.): Minimum value of the IRM, IRGM.
(17)
According to C, the quantitative and qualitative scales are established (table 8), where (x) represents the magnitude of IRM, IRGM.
Table No8: Qualitative evaluation scale for the IRM and IRGM values evaluated
Source: Self Made
The rest of the indices obtained by expressions 11, 12, 13, 14 are evaluated using the above table according to the approach they present (risk materialization or metrological management): IRM or IRGM, i.e., the scale allows for the same interval and qualitative assessment to obtain an evaluation for these indices.
The metrological indicators are the main source of information for evaluating the risk indices. The application of this by the companies should be annual, shortening or extending its evaluation frequency from certain events or situations within the business environment, so organizations should prepare their metrology specialists with the use of this tool or use consultants.
The analysis of the results of this evaluation allows to reach concrete and very useful information, from which conclusions can be drawn.
Once the indexes have been approved and the documentation for the collection of information has been defined, they are calculated in the aforementioned companies. The period in which the data will be taken for calculation must be decided. This stage is not dealt with in the research because it is still in the process of implementation.
CONCLUSIONS
Please refer to articles in Spanish Bibliography.
BIBLIOGRAPHICAL ABSTRACT
Please refer to articles Spanish Biographical abstract.