1. INTRODUCTION

The overall appraisal for a university teacher in a certain period is an important aspect in educational administration and management. An effective and workable comprehensive evaluation generally involves the consideration of several criteria which often further include more sub-criteria and then a two-level (or two-layer) criteria system ordinarily should be built before performing a thorough and comprehensive evaluation and the further decision making process. Multi-criteria decision-making (MCDM) have been systematically studied and been developed during the last decades [1–7], which proved to be one ideal, feasible and widely acceptable evaluation method.

In practice, the collected and involved information contain ever-increasingly more uncertainties. Uncertain information has a wide variety of different forms of uncertainties, such as probability information, fuzzy information and its extensions, linguistic information, hesitant information with some existing different types [8–11] and R-numbers [12]. Recently, researchers proposed to generalize some of the aforementioned uncertain information into a more common form, namely, the Basic Uncertain Information (BUI) [13,14]. This generalization can facilitate the handling of merging uncertain information with diverse types. Besides, BUI can quantify those uncertainties involved in different types of uncertain information, further providing some effective and convenient way to make judgment and decision according to whether some real-valued certainty degrees as thresholds have been attained.

It is appealing that a uniform and quantified certainty degree can be provided as an indication for decision makers to take judgments and decisions. Therefore, a comprehensive evaluation frame capable of collecting, transforming and melting certainty degrees is quite helpful in a large number of evaluation problems.

Against this background, to well perform a comprehensive MCDM problem in the environment of uncertain information, majorly we need to consider the following two aspects:

1. Weights allocation according to certainty degrees. Since in the evaluation process the involved uncertainty must be taken into consideration, some special information fusion mechanisms should also be reasonably and appropriately designed. Simply speaking, with the other conditions unchanged, if the information collected for one certain criterion has larger certainty, then it is reasonable to endow more weight to that certain criterion. This principle will be well performed and embodied using a series of preference involved weight allocation methods in this work.

2. Aggregation of certainty degrees. Aggregation operators [15] as some strict and powerful information merging techniques can be applied throughout the whole comprehensive evaluation problem. Aggregation operators have been studied and developed for long time [16–21], and they have ever-increasing applications and become a hot area of research [22–27] nowadays. To accommodate input vectors of BUI granules, some aggregation paradigms will be also proposed, which allows the aggregated result still to be BUI granules [13].

Based on the above two principles, this study will discuss and propose some reasonable aggregation model that can also take into consideration some initial and original given information and preference over the two layers of criteria used for evaluation. We will use the proposed model in university teacher performance appraisal, and all the individual evaluation information with different sub-criteria are collected and transformed into BUI.

The remainder of this study is organized as follows: Section 2 elaborately discusses and analyzes some uncertainties involved aggregation methods with some principles of weights generating and melting, which is instructive to the further model building. Section 3 proposes a series of detailed processes for the comprehensive evaluation with preferences and uncertainties. Section 4 provides a numerical example and application of the proposed evaluation model in educational management and evaluation.

2. THE UNCERTAINTIES INVOLVED AGGREGATION METHODS AND WEIGHTS DETERMINATION AND MELTING

In this section, we review the concept of BUI [13,14], discuss the induced weights allocation method for BUI vector and then discuss the corresponding weights melting and BUI aggregation, respectively.

We firstly fix some formulations related to the weighted averaging of real inputs. Without loss of generality, the input vector corresponding to n different criteria/sub-criteria is denoted by x=xjj=1n∈0,1n. The weighted averaging operator (with normalized weight vector w=wjj=1n∈0,1n, ∑j=1nwj=1) is a function WAw:0,1n→0,1 such that WAwx=∑j=1nwjxj.

BUI is a newly proposed generalization of some different types of uncertainty involved information. Without loss of generality, a BUI granule is formed by a pair <x;c> in which x∈0,1 is the data (evaluation value) pertaining to a concerned evaluation problem or single evaluation criterion, while c∈0,1 is the certainty degree of x, presenting the degree to which the involved decision makers are convinced that the data x is accurately, exactly or correctly collected, measured or recognized, and so on. As the complement, in a BUI granule <x;c> we commonly have the uncertainty degree for x by 1−c. Generally, the larger certainty degree c, the more confidence we have that the data x has represented the true value of some certain object, and vice versa. In a MCDM problem, if a BUI granule <x;c> with larger certainty degree c is obtained for a certain criterion, then we naturally should give more weight to that criterion.

Formally, a vector/sequence of n BUI granules is expressed either by <xj;cj>j=1n, or, with somewhat abuse [13], by <x;c>, wherein x=xjj=1n∈0,1n and c=cjj=1n∈0,1n. In addition, the space of all BUI granules is denoted by ℬ. With a given (normalized) weight vector w=wjj=1n, the weighted averaging operator with w for BUI [13] is defined to be a function WAw:ℬn→ℬ such that

For a collected vector of BUI granules <x;c>, it is desirable that the granule <xj;cj> with larger certainty degree cj should obtain more weight. The induced weights allocation mechanism using Regular Increasingly Monotonic (RIM) quantifier [28] can reasonably and efficiently fulfill this purpose. As we mentioned earlier, to model the desired preference for large certainty degrees, here we need a system of functions to serve as an embodiment or measurement of such preference. A RIM quantifier structurally is nothing but a nondecreasing function defined on unit interval Q:0,1→0,1 such that Q0=0 and Q1=1, and thus necessarily Riemann integrable from the knowledge of basic analysis. The integral related value, K=1−∫01Qxdx∈0,1, can be used as an indicator of the preference extent over the criteria that are associated with those BUI granules having higher certainty degrees; that is, when the value K is nearer 1, the preference is stronger for the information with high certainties, and vice versa. In this work, our concern is specifically on the system xtt∈1,+∞, in which a large t leads to a small integral value of xt and thereby a large extent of preference K over those information with high certainties. For example, when t=2, x2 leads to preference extent K=1−∫01x2dx=2/3, a moderate preference.

In detail, we have the following weights allocation method for a vector of BUI granules <xj;cj>j=1n, whose entries correspond to n different criteria Cjj=1n in some certain evaluation problem. Using the preference induced weight allocation techniques [28–30] together with the recently presented stricter three-set formulations [31], we firstly define three sets for each j∈1,…,n respectively:

Then, the allocated weight vector h=hjj=1n to the n criteria Cjj=1n is defined such that

where |A| represents the cardinality of any finite set A. It is noteworthy that in this system xtt∈1,+∞, since we do not allow to actually attain the very extreme preference, that is, t=+∞, then it is evident to observe that for each t∈1,+∞ the obtained h=hjj=1n always satisfies hj>0 for all j∈1,…,n.

With the obtained weight vector h=hjj=1n, if originally we have already assigned a weight vector w=wjj=1n to criteria Cjj=1n according to some other weight determination methods, then we need to melt these two weight vectors to obtain a resulted weight vector p=pjj=1n, for example, by the following way:

Note that since hj>0 for all j∈1,…,n, then the above formulation is meaningful because ∑k=1nhkwk>0 always holds. In addition, when t=1, indicating there is no preference involved, then we have h=hjj=1n=1/nj=1n, and thus p=w.

By the resulted weight vector p=pjj=1n, we have the following weighted aggregation process for BUI granules <xj;cj>j=1n=<x;c> with

The above formula will be applied several times in the later discussions and proposed models.

3. A DETAILED PROCESS FOR COMPREHENSIVE EVALUATION WITH PREFERENCES AND UNCERTAINTIES

In this section we present a comprehensive evaluation process with detailed procedures together with some comments when suitable. In practice, decision makers may freely change or modify some steps when necessary according to different environments and scenarios.

All the steps of the comprehensive evaluation with preference and uncertainties are subsumed into the following four major stages:

Stage 1 Criteria Building and their relative importance analysis

Step 1 Build a set of m first level evaluation criteria Ckk=1m.

Step 2 Build m sets of second level of evaluation criteria cjkj=1nk k=1,…,m.

Step 3 Determine a weight vector w=wkk=1m using AHP or some other weighting methods such as inviting experts to subjectively weight each criterion in the first level.

Step 4 For each k∈1,…,m determine a weight vector wk=wjkj=1nk using some weighting methods such as AHP to weight each criterion in the second level.

Stage 2 For each criterion/sub-criterion collect uncertain information

Step 1 Design a 5-scaled linguistic inquiry including (1 Extremely good; 0.8 Very good; 0.5 Good; 0.25 Not good; 0 Bad) for collecting evaluation of the evaluation object. (The values before the varying linguistic evaluations (i.e., 1, 0.8, 0.5, 0.25, 0) can be changed with different preferences or experiences of decision makers.)

Step 2 Design a 5-scaled linguistic inquiry including (1 Almost sure; 0.8 Sure; 0.5 Possibly; 0.3 Probably/Maybe; 0.1 Not sure) for collecting certainty degrees of their evaluations. (The values before the varying certainty degrees (i.e., 1, 0.8, 0.5, 0.3, 0.1) can also be changed according to the preferences of decision makers.)

Step 3 Distribute specifically designed questionnaires, inquiring respondents to provide with their individual evaluation about an evaluation object and also provide their own certainty degrees using the linguistic inquiry sheets.

Step 4 For each k∈1,…,m and each j∈1,…,nk, obtain two normalized statistic vector sjk=sj1k,sj2k,sj3k,sj4k,sj5k for evaluation values and ljk=lj1k,lj2k,lj3k,lj4k,lj5k for certainty degrees. (If for some k∈1,…,m and some j∈1,…,nk, the individual evaluation information is already with a BUI form <xjk;cjk>, then Step 1, 2 and 4 can be skipped.)

Step 5 Accumulate the collected information and take the averages of both information including evaluation values and the certainty information, and transform them into a BUI granule. In detail, for each k∈1,…,m and each j∈1,…,nk, obtain BUI vector <xjk;cjk>j=1nk by

xjk=sjk⋅1,0.8,0.5,0.25,0T and cjk=ljk⋅1,0.8,0.5,0.3,0.1T; or using an alternative expression, write <xjk;cjk>j=1nk as <xk;ck> where xk=xjkj=1nk and ck=cjkj=1nk.

Stage 3 Second level criteria weights melting and regeneration, and the aggregation for BUI vectors

Step 1 Select a RIM quantifier Q, for example, with Qx=x2 embodying some certainty inclined preference.

Step 2 For each k∈1,…,m and each j∈1,…,nk, define three sets Ujk=q|cqk>cjk, Ljk=q|cqk<cjk, Ejk=q|cqk=cjk.

Step 3 For each k∈1,…,m, generate a weight vector hk=hjkj=1nk in the following way:

Step 4 For each k∈1,…,m, melt hk with wk to yield a new weight vector pk=pjkj=1nk by

Step 5 For each k∈1,…,m, using pk to aggregate BUI vector <xk;ck>=<xjk;cjk>j=1nk and obtain the comprehensive evaluation results for the first level criteria Ckk=1m with

Stage 4 First level criteria weights melting and regeneration to obtain the aggregation for final judgment

Step 1 Form a certainty vector with respect to the first level criteria, that is, a vector of certainty degrees c=ckk=1m where ck=WApkck. Besides, denote x=xkk=1m where xk=WApkxk.

Step 2 For each k∈1,…,m, define three sets Uk=q|cq>ck, Lk=q|cq<ck, Ek=q|cq=ck.

Step 3 Generate a first level weight vector h=hkk=1m in the following way:

Step 4 Deal with the first level melting for h and w to yield a new first level weight vector p=pkk=1m by

Step 5 Applying p to aggregate BUI vector <x;c>=<xk;ck>k=1m and obtain the final comprehensive evaluation result:

4. AN ILLUSTRATIVE EXAMPLE IN EDUCATIONAL MANAGEMENT AND APPRAISAL

This section illustrates the proposed model in the preceding section with a numerical example in educational management and evaluation.

The object under evaluation is supposed to be a teacher in one university. His comprehensive performance over the last three years has indicative usage and is helpful for the management to evaluate and give some possible promotion to him, and the evaluation result may also help him to recognize his past working status in order to make improvement in future. In this study, after analyzing we have listed the following 3 main criteria together with 10 sub-criteria for evaluating a university teacher's working performance. Together with all the two-layer criteria, the approaches for obtaining individual evaluation of each sub-criterion is organized and listed in Table 1.

The detailed evaluation process is presented in what follows:

Stage 1 The criteria and sub-criteria with the original weight for each criterion are offered by experienced experts.

There are 3 first level evaluation criteria Ckk=13 and an original first layer weight vector w=0.2,0.3,0.5 is provided. For each k∈1,2,3, the three collections of sub-criteria are also listed as cj1j=13, cj2j=14, cj3j=13 with their respective second layer weight vectors w1=0.4,0.4,0.2, w2=0.25,0.25,0.25,0.25, w3=0.4,0.4,0.2.

Stage 2 For each criterion/sub-criterion collect uncertain information

A 5-scaled linguistic inquiry with scale values including (1 Extremely good; 0.8 Very good; 0.5 Good; 0.25 Not good; 0 Bad) and a 5-scaled linguistic inquiry with scale values including (1 Almost sure; 0.8 Sure; 0.5 Possibly; 0.3 Probably/Maybe; 0.1 Not sure) are used for collecting some subjective information, which includes the evaluation obtaining approaches of “Experts Appraising,” “Self Appraising” and “Students Responses.” For the sub-criteria that use “Objective Data,” directly obtain BUI granules as the individual judgments.

Managements distribute specifically designed questionnaires to experts, the teacher himself, and his students, and collect their judgments with certainty degrees using the linguistic inquiry sheets. Then, transform those linguistic evaluations with uncertainties into BUI granules, which are listed below. Note that all the judgments from “Objective Data” are assumed to be with certainty degree 1.

<x1;c1> is with x1=0.6,0.9,0.4 and c1=0.7,0.8,0.6;

<x2;c2> is with x2=1,0.7,0.8,0.8 and c2=1,0.5,1,0.5;

<x3;c3> is with x3=0.9,0.7,0.8 and c3=1,0.7,0.5

Stage 3 Second level criteria weights melting and regeneration, and the aggregation for BUI vectors.

Adopt a RIM quantifier Q such that Qx=x2 embodying a moderate certainty inclined preference.

Define sets U11=2, L11=3, E11=1; U21=∅, L21=1,3, E21=2;U31=1,2, L31=∅, E31=3.

U12=∅, L12=2,4, E12=1,3; U22=1,3, L22=∅, E22=2,4;U32=∅, L32=2,4, E32=1,3; U42=1,3, L42=∅, E42=2,4.

U13=∅, L13=2,3, E13=1; U23=1, L23=3, E23=2;U33=1,2, L33=∅, E33=3.

For each k∈1,2,3, generate a weight vector hk=hjkj=1nk using formula (3.1):

h1=1/3,5/9,1/9; h2=3/8,1/8,3/8,1/8; h3=5/9,1/3,1/9.

Then, for each k∈1,2,3, melt hk with wk to yield a new weight vector pk=pjkj=1nk using formula (3.2):

p1=6/17,10/17,1/17; p2=3/8,1/8,3/8,1/8; p3=10/17,6/17,1/17.

Finally, for each k∈1,2,3, using pk to aggregate BUI vector <xk;ck>=<xjk;cjk>j=1nk and obtain the comprehensive evaluation results for first level criteria Ckk=13 using formula (3.3):

WAp1<x1;c1>≐<0.765;0.753>; WAp2<x2;c2>=<0.8625;0.875>; WAp3<x3;c3>≐<0.824;0.865>.

Stage 4 First level criteria weights melting to obtain the aggregation for final overall judgment of that university teacher

Form the certainty vector with respect to the first level criteria c=0.753,0.875,0.865 and obtain the corresponding value vector x=0.765,0.8625,0.824.

Define sets U1=2,3, L1=∅, E1=1; U2=∅, L2=1,3, E2=2; U3=2, L3=1, E3=3.

Generate the first level weight vector h=1/9,5/9,1/3 using formula (3.4). Then, melting h and w to generate the new first level weight vector p=1/16,15/32,15/32 using formula (3.5). Finally, applying p to aggregate BUI vector <x;c> and obtain the final comprehensive evaluation result with formula (3.6).

With the above obtained final aggregation result, decision makers may freely take decisions depending on the detailed situations of them. Decision makers judge whether or not the final comprehensive aggregation result can help make overall judgment by some previously predetermined thresholds, which can be also a BUI granule. In this example, we set the thresholds to present some detailed linguistic judgments as follows:

If x*≥0.8 and c*≥0.65, then the teacher's performance is “Excellent.”

If x*<0.5 and c*≥0.85, then the teacher's performance is “Substandard”; else, the teacher's performance need to be reevaluated using some new collected information.

Clearly, in this case the teacher's performance is “Excellent.” Note that we choose a bigger value 0.85 for an indication of being “Substandard” is because to state one's negative performance generally it needs more convinced and safe information to testify. In some similar way, decision makers can design different thresholds according to their own situations.

We finally discuss some possible shortcoming of the proposed method. It is possible that some initially determined thresholds cannot help lead to conclusive evaluation and decision, and this situation may occur several times even if we readjust the thresholds every time. Therefore, in some extreme environment (e.g., in some environment where most of the data collected are with huge uncertainty and it is hard or costly to recollect new data with large certainty) the proposed model may fail and thus it is needed to devise some further adapted and improved models to address this problem.

5. CONCLUSIONS

One most featured advantage of using BUI in decision-making lies in that it can well generalize and quantify different types of uncertainty information. With this special property, some information fusion techniques designed for melting both input values and their attached certainty degrees becomes possible. This study proposed an effective and flexible comprehensive evaluation model which can simultaneously consider the original preference over different criteria and the new preference over the certainty degrees of the input information.

The proposed model with four major stages adopts two layers of criteria used for comprehensively evaluate a certain object. This model firstly considers the inner layer of criteria and obtains some intermediately aggregated BUI, and then performs the outer layer aggregation to obtain an overall resulted BUI to help make further judgments and decisions. By modifying or changing some single steps, decision makers can freely design some different merging schemes; for example, with using a combination method, some desired monotonicities can be also guaranteed.

A numerical application in educational evaluation about appraising a university teacher is proposed to illustrate the whole evaluation processes and to verify the feasibility of the proposed model. In addition, the techniques and theories of aggregating certainty degrees synchronized with merging input information are more general and have significant theoretical values in the study of aggregation operators and information fusion.

CONFLICTS OF INTEREST

The authors declare they have no conflicts of interest.

AUTHORS' CONTRIBUTIONS

All authors contributed to the work. All authors read and approved the final manuscript.

ACKNOWLEDGMENTS

This work is partly supported under Scientific Research Start-up Foundation with Grant 184080H202B165. This work is partly supported by the National Natural Science Foundation of China (grant no. 71801175).