Throwing out zeros from a faulty sensor

P

pteehan

New Member
#1
#1
I'm fairly new at stats. I have time-series data from a sensor. There are a lot of zeros in the data -- 70% of readings -- and those seem improbable physically. By visual inspection they seem to be faults, but the data otherwise looks like I expect it to. I plan to just throw away the zeros and interpolate those points. But it is possible (though unlikely) that there are some legitimate zeros being measured. You can look at the data here: http://imgur.com/RIlOpLt,al6dFW9

Intuitively it seems very improbable that the data would jump to zero, but I would like to be able to demonstrate this with some level of confidence and don't know where to start. If I plot P(i) against P(i+1) (for example) there is a clear gap between the zero and non-zero points. But what if the gap was small or a bit fuzzy making zero seem more plausible? How could I differentiate between spurious zeros and legitimate ones?
 
Miner

Miner

TS Contributor
#2
#2
You may be dealing with a "limit of detection" issue with your sensor. That is, when the factor that you are measuring falls below a certain level yet is still non-zero, the sensor can no longer detect it and provides a zero output.
 
hlsmith

hlsmith

Less is more. Stay pure. Stay poor.
#4
#4
Most products may have available documentation on the accuracy of the technology, in that it was quality tested internally to know its shortcomings. You may benefit from researching the sensor to fully understand how it performs and if as Miner references, there may be a level of inability to discern or quantify traceable amounts.

What are your plans with this data? If publishing, authors will list the exact equipment used so others can attempt to replicate their experiment. If deciding to throw something out you need to be thorough in your description of your methods so others can fully understand your results.
 
You must log in or register to reply here.