SPPA Scoring

From Dan Aloi- University of Connecticut School of Medicine

Q: How is the SPPA calculated?

A:

fig. 1

Inter-rater agreement

Item

First, you need to calculate the Difference values:

• For an Item, calculate the difference between the value of the answer selected by the SP and the one selected by the Observer.

sum|O-SP|

O = the value of the answer selected by the observer for an item

SP = the value of the answer selected by the SP for an item

Item, if the SP & Observer agree and select the same answer, the difference value for this item is 0.

Item calculation: 1-1= 0

fig 2.

yes=1pt no=0pts

Item, if the SP & Observer disagree and select different answers, the difference value for this item is 1.

Item calculation: 1-0=1

fig 3.

yes=1pt no=0pts

If the item is a multiple choice answer type (for example checkbox), you need to sum the values of the answers of the SP for the given item. The same needs to be done with the answers of the Observer and calculate the absolute value of that difference.

Using the example (fig. 1) above, the Difference score is 3.

1. 1-1 = 0
2. 1-1 = 0
3. 1-1 = 0
4. 1-1 = 0
5. 1-1 = 0
6. 1-0 = 1
7. 1-0 = 1
8. 1-0 = 1
9. 0-0 = 0
10. 0-0 = 0

Section

Then, calculate the section agreement value

• For a Section, it is the total achievable points in that section and the difference value, divided by the total achievable points. *Note that this value is in points, not percentages.

((t-d)/t)*100

* t = total achievable point value in the section

* d = the difference value for the section

Using the example (fig. 1) above, and the Difference score of 3.

Section calculation: (10-3)/10)*100 = 70% for the History of Present Illness Section

Weights

WebSP also considers section weights. WebSP calculates the inter-rater agreement value using the section weights as set in the Case Manager for learner scoring. This ensures that disagreement about points in more important (higher weighted) sections will affect the total agreement value more.

The Observer inter-rater agreement value = sum of the products of each section agreement value within one particular SP part, and the corresponding section weight, divided by the sum of the section weights.

sum(s*w)/sum(w)

* s = the section agreement value for each section of the SP part of a certain case

* w = the section weight belonging to each section within the SP part, defined in the Case Manager

((70*100)+(86*100)+(95*100))/(100+100+100) = 83.67%

Multiple Observations/Inter-rater Agreements

SPs may be evaluated for several encounters. The total inter-rater agreement score is an average of the inter-rater agreement values from all observations of the SP for a certain case.

i/n

* i = the inter-rater agreement value of an SP, for one particular encounter (i.e. observation)

* n = the number of observations the SP has for a certain case

SP Performance Assessment

Sections

The SPPA sections are calculated similar to the way Learners are scored.

sum(s*w)/sum(w)

* s = the section score of an SP, for each section of the SPPA part of one particular case

* w = the section weight belonging to each section within SPPA part of the case, defined in the Case Manager

For example, let's say an SPPA has two sections

1. Inter-Rater Report Portrayal (I/R Portrayal)
2. Inter-Rater Report Feedback (I/R Feedback) section is 25 and the SP achieved 23.

The total achievable score in the Inter-Rater Report Portrayal (I/R Portrayal) section is 15 and the SP achieved 15.

The total achievable score for the Inter-Rater Report Feedback (I/R Feedback) section is 25 and the SP achieved 23.

Based on this, the calculation is the following:

(((15/15*100)*100)+((23/25*100)*100))/(100+100) = 96%

Multiple Observations/SPPA

SPs may be evaluated for several encounters. The total inter-rater agreement score is an average of the inter-rater agreement values from all observations of the SP for a certain case

sum(e)/n

* e = the SP performance evaluation value of an SP, for one particular encounter for each observation the SP has for a certain case

* n = the number of encounters that have been observed for the SP in the case

The Total SP Performance Score

The Total SP Performance Score is the weighted average of the inter-rater agreement value and the SP performance evaluation value.

(sp_part_wsum * agrval + spa_part_wsum * sppescore)/(sp_part_wsum + spa_part_wsum)

* sp_part_wsum = the sum of the weights of the SP part

* agrval = the inter-rater agreement value of the SP for the encounter

* spa_part_wsum = the sum of the weights of the SPPA part

* sppescore = the SP performance evaluation value for one particular encounter of an SP

((100+100+100)*70)+((100+100)*96)/((100+0+100+100)+(100+10 0)) = 91%

Overall SP Performance Score

The overall performance of the SP is indicated with the Total SP Performance Score, which is the average of the inter-rater agreement score and the SP Performance Assessment score.

sum(t)/n

* t = the total SP performance value of an SP, for one particular encounter for each observation the SP has for a certain case

* n = the number of encounters that have been observed