Mixing
How to normalize noisy data?
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I'm a computer science student and for a project I'm taking sensor data and normalizing it into a 0 - 255 range. Problem is when I get the data I don't know the minimum and maximum (it varies very widely based on where the sensor is located) and that really affects how normalizing the data goes. I would just find the min and max in the data set but it's noisy enough outliers screw this up. This image isn't my data, but the graph looks similar.
My question is how to find a reasonable minimum and maximum to normalize against. I was thinking about finding the average minimum (average of all the lows) and average maximum (average of all the highs) and using those, but I'm not sure how I would go about doing that in an efficient way. Currently I'm graphing the data in excel and deciding a reasonable minimum and maximum but that isn't feasible in the long term. For the graph I linked, I would probably use a min of 190 and a max of 270 to normalize it.
Thanks for any help!
probability statistics data-analysis
asked Jul 23 at 18:40
lilibug1
91
add a comment |Â
up vote
1
down vote
favorite
I'm a computer science student and for a project I'm taking sensor data and normalizing it into a 0 - 255 range. Problem is when I get the data I don't know the minimum and maximum (it varies very widely based on where the sensor is located) and that really affects how normalizing the data goes. I would just find the min and max in the data set but it's noisy enough outliers screw this up. This image isn't my data, but the graph looks similar.
My question is how to find a reasonable minimum and maximum to normalize against. I was thinking about finding the average minimum (average of all the lows) and average maximum (average of all the highs) and using those, but I'm not sure how I would go about doing that in an efficient way. Currently I'm graphing the data in excel and deciding a reasonable minimum and maximum but that isn't feasible in the long term. For the graph I linked, I would probably use a min of 190 and a max of 270 to normalize it.
Thanks for any help!
probability statistics data-analysis
asked Jul 23 at 18:40
lilibug1
91
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm a computer science student and for a project I'm taking sensor data and normalizing it into a 0 - 255 range. Problem is when I get the data I don't know the minimum and maximum (it varies very widely based on where the sensor is located) and that really affects how normalizing the data goes. I would just find the min and max in the data set but it's noisy enough outliers screw this up. This image isn't my data, but the graph looks similar.
My question is how to find a reasonable minimum and maximum to normalize against. I was thinking about finding the average minimum (average of all the lows) and average maximum (average of all the highs) and using those, but I'm not sure how I would go about doing that in an efficient way. Currently I'm graphing the data in excel and deciding a reasonable minimum and maximum but that isn't feasible in the long term. For the graph I linked, I would probably use a min of 190 and a max of 270 to normalize it.
Thanks for any help!
probability statistics data-analysis
asked Jul 23 at 18:40
lilibug1
91
I'm a computer science student and for a project I'm taking sensor data and normalizing it into a 0 - 255 range. Problem is when I get the data I don't know the minimum and maximum (it varies very widely based on where the sensor is located) and that really affects how normalizing the data goes. I would just find the min and max in the data set but it's noisy enough outliers screw this up. This image isn't my data, but the graph looks similar.
My question is how to find a reasonable minimum and maximum to normalize against. I was thinking about finding the average minimum (average of all the lows) and average maximum (average of all the highs) and using those, but I'm not sure how I would go about doing that in an efficient way. Currently I'm graphing the data in excel and deciding a reasonable minimum and maximum but that isn't feasible in the long term. For the graph I linked, I would probably use a min of 190 and a max of 270 to normalize it.
Thanks for any help!
probability statistics data-analysis
asked Jul 23 at 18:40
lilibug1
91
asked Jul 23 at 18:40
lilibug1
91
asked Jul 23 at 18:40
lilibug1
91
asked Jul 23 at 18:40
lilibug1
91
91
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17
add a comment |Â
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17
add a comment |Â
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2860655%2fhow-to-normalize-noisy-data%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
en.wikipedia.org/wiki/Smoothing
– saulspatz
Jul 23 at 18:46
I don't see that this plot is very noisy (about 5% amplitude) so it is hard to understand what you need.
– Yves Daoust
Jul 23 at 19:23
Do you have a histogram of the data? It seems like you spend enough time near the minima and maxima that Tukey's fences aren't great.
– Brian Tung
Jul 23 at 22:17