Thread: Performance of CSIs at Crime Scenes

1. Performance of CSIs at Crime Scenes

Hi,

Im after some advice, which i would really appreciate assistance with.

At our police force the CSIs are being performance tested by use of statistics. The statistics reveal the following (per 3 months):

(1) How many fingerprint/DNA matches the CSI has gained (linking the crime scene sample to the offender)
(2) The number of crime scenes attended
(3) The number of exhibits seized

Then calculated is
(1) The average fingerprint/DNA identifications for the office/individual - then each CSIs identifications are compared to the office average.
(2) The percentage of fingerprint/DNA identifications from the total number of exhibits seized

The problem I have with these statistics is that
(1) Not all crime scene are the same, there are so many vairables, one might attend serval burglary scenes and find glove marks left at the scene
(2) There is an assumption that fingerprint/DNA identifications are equally spread across every CSI in the office.
(3) Not all crime scenes render evidence - shed/garage burglary will have little to nothing evidentially, wheras a burglary may have much more due to the nature of the incident
(4) The jobs we attend are not equally distributed, we are dispatched to jobs as, and when they become available, and what scene your given is not your choice.

I think there are several more 'assumptions' associated with these calculations. Id be grateful is you could advise on your views please.

This is a new thing which is being done at the police force which i have a problem with, it just doesnt seem fair to me, as there are so many variables.

We are trying to writeup a statement to management to say that these stats are not the way forward. The reason im writing here is if you can assist in sharing your stats knowledge which may give us more ammunition to show how skewed the stats calculations are.

Thank you

2. Re: Performance of CSIs at Crime Scenes

hi,
there us a deeper problem with the method as you explained it . We sometimes define performance measures for companies and the one warning / question we always give them is: would it be good for the company if people changed their behavior in a way to optimize these measures?

E.g if I were measured on the number of identifications, I would definitely avoid difficult cases like the pest.

My suggestion would then be to think about the behavior changes this measurement would bring about and argue from that angle .

disclaimer This is definitely not to say that measurements are bad, they just need to be very carefully designed.

good luck
rogojel

3. Re: Performance of CSIs at Crime Scenes

Originally Posted by rogojel
hi,
there us a deeper problem with the method as you explained it . We sometimes define performance measures for companies and the one warning / question we always give them is: would it be good for the company if people changed their behavior in a way to optimize these measures?

E.g if I were measured on the number of identifications, I would definitely avoid difficult cases like the pest.

My suggestion would then be to think about the behavior changes this measurement would bring about and argue from that angle .

disclaimer This is definitely not to say that measurements are bad, they just need to be very carefully designed.

good luck
rogojel

I really appreciate your suggestion, it was something I didnt think about but is a very valid point and a way to approach the problem.

I would like to approach the statistical side of things too, from looking at the points I made in the original post - is there any points I need to make to show that the statistics are wrong, invalid and flawed to judge the performance of a CSI? - I assume you work in the field os statistics so your views will be very, very much appreciated.

Thanks again

4. Re: Performance of CSIs at Crime Scenes

Hi,
I am a bit wary about going into the details because I have no practical knowledge of the field, but maybe a few more thoughts:

- another question to ask would be imho the goal of the measurement, the way this data is intended to be used. E.g. if this is to rank the people - it should be noted that at any given time roughly 50% of the force will be below average (a well known statistics joke in fact). For ranking purposes all the confounding factors you listed, and more, will be active - so the simple average is a singularly ill chosen metrics for that. In fact, given the possibilities to game the system I would just not use any measurement for ranking.

In the book "Thinking like a Freak" there is some good advice on this:

1. Create incentives that switch the frame from adversarial to cooperative
2. Never ever think people will do something just because it is the right thing to do
3. Know that some people will do everything to game the system finding ways to win that you never could have imagined.

- if the goal is to gain an overall view of the units performance then it would make a lot of sense to anonymize the measurements (as no one is interested in individual performance, right?) and use the right type of control chart . It would have the advantage of providing the information without the incentive to game the system. Just to gain more insight I would gather the factors that influence the metrics (e.g. a rating of the site on a Likert scale (very Easy, easy, average, Difficult , Very difficult) and record that information and evrything else you come up with, too.

So, I would really discuss this from the angle of cooperation - can the measurements be used in a way to help the employees? Are they used that way? If yes on both counts, then you can (will) go into the discussion of how to frame them so that their efficiency is maximized.

I know this is not exactly what you expect, but maybe it will help you a bit?

regards
rogojel

 Tweet

Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts