Mixing
Determining Significance in Trends
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I am responsible for metrics reporting within our IT organization. This involves monthly tracking of various metrics - for this post I will use one example: Incident Count (a.k.a Ticket Count). Each metric has a monthly count and a yearly avearage (the average of all of the months YTD). Here is a a real-life example:
Jan 2018: 4881
Feb 2018: 4502
Mar 2018: 5255
Apr 2018: 5310
May 2018: 5350
Jun 2018: 4576
Jul 2018: 4999
2018 YTD Monthly Avg: 4982
We want to add some up / down arrow indicators to show the trend direction. My boss has asked that these indicate some sort of significant trend. In other words, he doesn't want it to show that the trend is down if we have gone from 4,999 Incidents to 5,000 from one month to another. I thought about using a +/- standard deviation, maybe a certain percent, but I keep thinking there is a better way. I'd like to use the same methodology across all of our metrics, so the stats/math do not change, but work for metrics with any type of count or variance.
Some things that I can point out - since we are doing current year monthly numbers, our sample size is small (1-12 months), some metrics have larger numbers (4,000+) and others have smaller numbers (~100 or so). I feel like there should be a way to determine significant change (statistically) for a data set like this. Keep in mind, all of our metrics will be the same sample size (1-12 months). Are there any basic methods I can use that I can use to determine if my trend is truly going up or down in a significant way (other than just comparing two months). I am coding this up in R, and just want something standard I can use across the board and is reproducible. Thanks!
statistics standard-deviation
asked Aug 6 at 23:04
azdatasci
1
 |Â
show 1 more comment
up vote
0
down vote
favorite
I am responsible for metrics reporting within our IT organization. This involves monthly tracking of various metrics - for this post I will use one example: Incident Count (a.k.a Ticket Count). Each metric has a monthly count and a yearly avearage (the average of all of the months YTD). Here is a a real-life example:
Jan 2018: 4881
Feb 2018: 4502
Mar 2018: 5255
Apr 2018: 5310
May 2018: 5350
Jun 2018: 4576
Jul 2018: 4999
2018 YTD Monthly Avg: 4982
We want to add some up / down arrow indicators to show the trend direction. My boss has asked that these indicate some sort of significant trend. In other words, he doesn't want it to show that the trend is down if we have gone from 4,999 Incidents to 5,000 from one month to another. I thought about using a +/- standard deviation, maybe a certain percent, but I keep thinking there is a better way. I'd like to use the same methodology across all of our metrics, so the stats/math do not change, but work for metrics with any type of count or variance.
Some things that I can point out - since we are doing current year monthly numbers, our sample size is small (1-12 months), some metrics have larger numbers (4,000+) and others have smaller numbers (~100 or so). I feel like there should be a way to determine significant change (statistically) for a data set like this. Keep in mind, all of our metrics will be the same sample size (1-12 months). Are there any basic methods I can use that I can use to determine if my trend is truly going up or down in a significant way (other than just comparing two months). I am coding this up in R, and just want something standard I can use across the board and is reproducible. Thanks!
statistics standard-deviation
asked Aug 6 at 23:04
azdatasci
1
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47
 |Â
show 1 more comment
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am responsible for metrics reporting within our IT organization. This involves monthly tracking of various metrics - for this post I will use one example: Incident Count (a.k.a Ticket Count). Each metric has a monthly count and a yearly avearage (the average of all of the months YTD). Here is a a real-life example:
Jan 2018: 4881
Feb 2018: 4502
Mar 2018: 5255
Apr 2018: 5310
May 2018: 5350
Jun 2018: 4576
Jul 2018: 4999
2018 YTD Monthly Avg: 4982
We want to add some up / down arrow indicators to show the trend direction. My boss has asked that these indicate some sort of significant trend. In other words, he doesn't want it to show that the trend is down if we have gone from 4,999 Incidents to 5,000 from one month to another. I thought about using a +/- standard deviation, maybe a certain percent, but I keep thinking there is a better way. I'd like to use the same methodology across all of our metrics, so the stats/math do not change, but work for metrics with any type of count or variance.
Some things that I can point out - since we are doing current year monthly numbers, our sample size is small (1-12 months), some metrics have larger numbers (4,000+) and others have smaller numbers (~100 or so). I feel like there should be a way to determine significant change (statistically) for a data set like this. Keep in mind, all of our metrics will be the same sample size (1-12 months). Are there any basic methods I can use that I can use to determine if my trend is truly going up or down in a significant way (other than just comparing two months). I am coding this up in R, and just want something standard I can use across the board and is reproducible. Thanks!
statistics standard-deviation
asked Aug 6 at 23:04
azdatasci
1
I am responsible for metrics reporting within our IT organization. This involves monthly tracking of various metrics - for this post I will use one example: Incident Count (a.k.a Ticket Count). Each metric has a monthly count and a yearly avearage (the average of all of the months YTD). Here is a a real-life example:
Jan 2018: 4881
Feb 2018: 4502
Mar 2018: 5255
Apr 2018: 5310
May 2018: 5350
Jun 2018: 4576
Jul 2018: 4999
2018 YTD Monthly Avg: 4982
We want to add some up / down arrow indicators to show the trend direction. My boss has asked that these indicate some sort of significant trend. In other words, he doesn't want it to show that the trend is down if we have gone from 4,999 Incidents to 5,000 from one month to another. I thought about using a +/- standard deviation, maybe a certain percent, but I keep thinking there is a better way. I'd like to use the same methodology across all of our metrics, so the stats/math do not change, but work for metrics with any type of count or variance.
Some things that I can point out - since we are doing current year monthly numbers, our sample size is small (1-12 months), some metrics have larger numbers (4,000+) and others have smaller numbers (~100 or so). I feel like there should be a way to determine significant change (statistically) for a data set like this. Keep in mind, all of our metrics will be the same sample size (1-12 months). Are there any basic methods I can use that I can use to determine if my trend is truly going up or down in a significant way (other than just comparing two months). I am coding this up in R, and just want something standard I can use across the board and is reproducible. Thanks!
statistics standard-deviation
asked Aug 6 at 23:04
azdatasci
1
asked Aug 6 at 23:04
azdatasci
1
asked Aug 6 at 23:04
azdatasci
1
asked Aug 6 at 23:04
azdatasci
1
1
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47
 |Â
show 1 more comment
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47
 |Â
show 1 more comment
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2874393%2fdetermining-significance-in-trends%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
You might be interested in a control chart: en.wikipedia.org/wiki/Control_chart If you google "control chart in r" you will find information on R packages for control charts.
– awkward
Aug 7 at 15:06
@awkward - Good stuff, I used this and I like it. I Think for now, to keep it simple, I'm going to use a % for all of my metrics so I can make it the same across the board, then as I move down the road, I can incorporate some more statistical elegance into this for each metric....
– azdatasci
Aug 7 at 16:45
Just a word of caution, it may not be good to base your control chart on percentages unless the control chart is designed for that purpose (in which case it may say something about "proportions"). Sometimes the control charts are designed to be used for "number of occurrences" of something, and if so that's how they should be used.
– awkward
Aug 7 at 17:03
@awkward - Oh, just to clarify, I didn't mean I'd use it on % values, I meant, that for the time being, to determine a significant change, I'm going to look at the current and previous month's % difference. If it's +/- 3%, I'll just label it as an up or down trend. Anything inside of that will be considered no significant change. For now, this will work and I can take more time to wrap my head around using a real statistical method to determine significance.
– azdatasci
Aug 7 at 17:36
OK, the important thing is that each type of control chart comes with a set of assumptions and limitations, and you want to use it within those constraints.
– awkward
Aug 7 at 17:47