Best/fastest way to detect invalid value?

[hyperion007]

Just thought I would ask for your opinions on how to detect an invalid value.

Example:

Reading temperature values from a sensor with 5 second intervals and storing them into three variables.

Var1
Var2
Var3

(more readings with shorter intervals is not a problem)

Temperature variation between readings can be quite large. The problem is that sometimes the reading is corrupt or simply wrong. It may show:

Var1=23.023
Var2=1205.440
Var3=23.169

Var2 is obviously wrong and I can create some IF decisions to detect that. But what if Var1 or Var 3 is the incorrect one?

Basically I want to take a number of samples, reject any samples that are not anywhere near the rest of the samples and THEN calculate an average on the remaining, valid, samples.

Any ideas on how to do this would help.

Thanks.

[Rudi]

Just thought I would ask for your opinions on how to detect an invalid value.

Example:

Reading temperature values from a sensor with 5 second intervals and storing them into three variables.

Var1
Var2
Var3

(more readings with shorter intervals is not a problem)

Temperature variation between readings can be quite large. The problem is that sometimes the reading is corrupt or simply wrong. It may show:

Var1=23.023
Var2=1205.440
Var3=23.169

Var2 is obviously wrong and I can create some IF decisions to detect that. But what if Var1 or Var 3 is the incorrect one?

Basically I want to take a number of samples, reject any samples that are not anywhere near the rest of the samples and THEN calculate an average on the remaining, valid, samples.

Any ideas on how to do this would help.

Thanks.

Hi hyperion007,


my suggestion in you wrote "(more readings with shorter intervals is not a problem)"..

create more variables,

VAR1
VAR1a
VAR1b

VAR2
VAR2a
VAR2b

VAR3
VAR3a
VAR3b

and reference the a and b to the base variable with a "tolerance" examp. 5 % .. or how fine tuning the temp is..
and pick the right variable for yours. the interval i would take short time... no idea .. is all 3 sec ok? test it..

(*
all 5 sec for one value?
so you can make interval for reference Value
0
VAR1 ...
in 1.5 s
VAR1a ...
in 1.5 s
VAR1b
3
you have 2 sec for caclulate VAR1

0
VAR2 ...
in 1.5 s
VAR2a ...
in 1.5 s
VAR2b
3
you have 2 sec for caclulate VAR2

0
VAR3 ...
in 1.5 s
VAR3a ...
in 1.5 s
VAR3b
3
you have 2 sec for caclulate...VAR3
*)


so you have three values from one Base Variable, one ore more is right..

if all three value wrong ... hmm ...

...second suggestion:

how much can the Temp be if right? "range limit"

you can in second or combinied calculate this with a test in a range limit.
max 2 digits...
max 80.00
min 0.00
and so on.

hope this help

,,an other:
...At last ... you can change the sensor



Best wishes
Rudi



edit... (*..*)
"with 5 second intervals "..
i have overredad this.. i edit the post at (*..*)

[Benj]

Hello,

Might be worth doing something like this but not sure it will work for every situation.

1) check the read is in a valid range, maybe -10 to 100
2) check if the read is + or - say 10 from the last average reading.
3) each read that is in range gets added to an accumulator and the number of passed samples gets incremented by 1.
4) at the end of sampling the sensor you divide the value in the accumulator variable by the number of passed samples to get the new average.
5) the accumulator and count value get reset ready for the next round.

A problem might be at startup where the last average value is not present. Maybe this could be less strict on the first pass so as to base the new average on any reading which is within the potentially valid range.

[hyperion007]

There might be another way of sorting this out. I have tried this with two sensors, one IR temperature sensor and one MAX31855 thermocouple sensor. The sensors and the read routine works just fine, no errors what so ever, when displaying the reading on a display as a string. But when I add the GSM component to the mix, it all seems to get mixed up. Even though I don't have any interrupts and the sensor reading routine has it's own macro, the macro somehow get's interrupted sometimes, not allways (frequency depends on the length of delay after sending data to the GSM module, to short and it always corrupts the sensor reading that follows)

I will most likely put the PIC into sleep mode and wake it once every 10 seconds, take a couple of readings, then start the GSM component and send the data. When done, put the PIC to sleep again. But if I would like to use the GSM component together with a GPS component for tracking movement then I would need an update speed that is as fast as possible and wouldn't want to use massive delays. But the question about how to detect invalid values is still interesting. Since I'm uploading my data to a MySQL database I can always use queries including commands like STD(temperature) to determine the standard deviation of for instance the 10 previous temperature readings and if the current value isn't within +/- 2*STD then it's invalid and will not be used.

[hyperion007]

I found this simple explanation of Mean Absolute Deviation / Median Absolute Deviation.
It will probably have to be implemented on the server side as I don't think an 8-bit PIC could perform these calculations (or would take way to long to do so)

http://www.graphpad.com/support/faqid/1299/

[Benj]

Hello,

It sounds like you have some memory corruption going on which is not a good sign.

In the GSM component what is the String Array Size and Message Array Size properties set to, these define the amount of memory to use to store the incoming data?

There is a similar setting for the GPS component.

Also check your config settings and ensure that extended CPU is set to disabled as this can cause problems with memory arrays.

Do you get any warnings during the compile?

[hyperion007]

Ah, that kind of makes sense. I'm at work now but from the top of my head I think that the Message Array Size property is set to 60 (default I think) but the actual string variable is set to String_Var[82]
I'll confirm when I get home.

Thanks.

[hyperion007]

Ok, so the String Array Size and Message Array Size are both set to 80. So was my variable called Data_String[80]

But aren't these for handling the incoming messages FROM the GSM module? And isn't it strange that increasing the delays after the last command sent to the GSM module which basically tells it to send the data I have already sent to the GSM module from the PIC, solves the problem of corrupt values almost completely?

For instance, when I tried using my IR temperature sensor before I increased the above mentioned delay, the readings that got uploaded to the database looked like this:

22.*/"//
22.-))/*

etc.

So the digits before the decimal came out correct but the rest was just scrambled, but did repeat itself so I guess the random ascii characters did represent actual numbers and was not completely random, but I can't be sure.

After increasing the delay, it worked fine, except maybe 5-10 times over a span of 3000 cycles.

[Benj]

Hello,

I have experienced before on an AVR where I2C and UART were being used at the same time the I2C data was somehow leaking onto the UART data and corrupting it. I ended up having to create a simple mutex in software so that the I2C would only ever be allowed to operate when the UART was inactive.

Might therefore be something due to a bug in the silicone for the chip, useful to look through the Errata doc on the microchip website to see what bugs if any they have found in the device.

The incoming GSM data is automatically appended to the end of the message array i.e. the echo from the command your sending out. It should look for the end of the array and not go beyond it but there is potential that one of the buffers is overflowing somewhere and leaking into the RAM space of another variable. Is it just the GSM component and the I2C component your using? If you can attach your program so far then we might be able to help spot something.

[hyperion007]

Hi Ben,

I'll send you the program file in a PM.

There is no macro for the GSM component to un-initialize it, as I found out when I thought I would just do that after the data was sent.

I have tried this exact program on both the PIC18F26K80 and PIC18F4550 with same results.

Thanks for checking my program for me!

[Benj]

Hello,

In your config settings turn off the "Extended Instruction Set" setting, this is known to do strange things with data arrays and fingers crossed this will fix the problems you've having.

[hyperion007]

I'll try again but when I tried it with PIC18F4550, exactly the same program, I had the "Extended Instruction Set" disabled.

Will let you know once I get home.

[hyperion007]

I guess you know your stuff I can now send data in 1,5-2 second intervals (might be able to push it higher but there's no real point in super fast updates using GPRS)

I usually don't forget to disable the "Extended Instruction Set" but sure enough

THANKS!


|date_record |temperature01

|2014-05-08 16:13:51 |22.109985

|2014-05-08 16:13:53 |22.049987

|2014-05-08 16:13:55 |22.009979

|2014-05-08 16:13:56 |22.130004

|2014-05-08 16:13:58 |22.229980

|2014-05-08 16:14:00 |22.350006

|2014-05-08 16:14:02 |22.009979

|2014-05-08 16:14:04 |22.069976

|2014-05-08 16:14:05 |22.169982

|2014-05-08 16:14:07 |22.009979

|2014-05-08 16:14:09 |21.889984

|2014-05-08 16:14:11 |21.789978

|2014-05-08 16:14:13 |22.130004

|2014-05-08 16:14:15 |22.250000

|2014-05-08 16:14:16 |21.910003

|2014-05-08 16:14:18 |22.109985

|2014-05-08 16:14:20 |22.049987

|2014-05-08 16:14:22 |22.109985

|2014-05-08 16:14:23 |22.049987

|2014-05-08 16:14:25 |22.109985

|2014-05-08 16:14:27 |22.229980

|2014-05-08 16:14:29 |21.949981

|2014-05-08 16:14:32 |21.889984

|2014-05-08 16:14:38 |22.069976

|2014-05-08 16:14:39 |21.729980

|2014-05-08 16:14:41 |22.350006

|2014-05-08 16:14:43 |22.049987

|2014-05-08 16:14:45 |22.130004