Mixing
![What is the way to Liberation: direct communication with lord Krishna or the reading scriptures? [on hold]](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZCTB4-bb-s-32SAm6ytjzFOU06cH9o8q-a_wq_UnH6lcd8hvws23CmBWHM6g256LzTX9yK1EiCCzTC1NSgtxlNk_J9hdr9bA3tD0Nqntbl521j4bLAm_uXPANuWLBhcCKTKzns2gaUodx/s72-c/1.jpg)
How to evaluate the change of information of a random variable?
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
Given a random variable $X$ having finite alphabet $mathcalA_X$ and valid $p_x(cdot)$ (for which there is no $x_0 in mathcalA_X$ so that $p_x(x_0) = 0$) I want to know its actual outcome (speaking about sport, $X$ could be the team that would buy a really famous player: the experts would give me $mathcalA_X$ and $p_x(cdot)$).
I'm willing to pay a $V$ amount of money for this information and I've found someone who could help me. Before the closing of the deal I find out that the outcome of $X$ won't surely be, let's say, $x_0$.
Since prior this discovery the value of $X$ was $V$, what is the value $V^star$ I should pay for the information?
How I would do - I would evaluate $H(X)$ using the definition, then I'd declare a new random variable $Y$ so that $mathcalA_Y = mathcalA_X ,/, x_0$ and $p_Y(y) = fracp_x(y)1 - p_x(x_0)$ and finally I'd evaluate $H(Y)$. Now I guess that $V^star = fracH(Y)H(X)V$.
I think this approach could be valid but I am not sure about that since it would be more natural to me to evaluate something like $H(X ,|, X neq x_0)$. Also, I'm not sure that a proportion is the way to go for finding $V^star$.
random-variables entropy
asked Jul 28 at 18:22
Giulio Scattolin
15619
 |Â
show 2 more comments
up vote
0
down vote
favorite
Given a random variable $X$ having finite alphabet $mathcalA_X$ and valid $p_x(cdot)$ (for which there is no $x_0 in mathcalA_X$ so that $p_x(x_0) = 0$) I want to know its actual outcome (speaking about sport, $X$ could be the team that would buy a really famous player: the experts would give me $mathcalA_X$ and $p_x(cdot)$).
I'm willing to pay a $V$ amount of money for this information and I've found someone who could help me. Before the closing of the deal I find out that the outcome of $X$ won't surely be, let's say, $x_0$.
Since prior this discovery the value of $X$ was $V$, what is the value $V^star$ I should pay for the information?
How I would do - I would evaluate $H(X)$ using the definition, then I'd declare a new random variable $Y$ so that $mathcalA_Y = mathcalA_X ,/, x_0$ and $p_Y(y) = fracp_x(y)1 - p_x(x_0)$ and finally I'd evaluate $H(Y)$. Now I guess that $V^star = fracH(Y)H(X)V$.
I think this approach could be valid but I am not sure about that since it would be more natural to me to evaluate something like $H(X ,|, X neq x_0)$. Also, I'm not sure that a proportion is the way to go for finding $V^star$.
random-variables entropy
asked Jul 28 at 18:22
Giulio Scattolin
15619
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05
 |Â
show 2 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Given a random variable $X$ having finite alphabet $mathcalA_X$ and valid $p_x(cdot)$ (for which there is no $x_0 in mathcalA_X$ so that $p_x(x_0) = 0$) I want to know its actual outcome (speaking about sport, $X$ could be the team that would buy a really famous player: the experts would give me $mathcalA_X$ and $p_x(cdot)$).
I'm willing to pay a $V$ amount of money for this information and I've found someone who could help me. Before the closing of the deal I find out that the outcome of $X$ won't surely be, let's say, $x_0$.
Since prior this discovery the value of $X$ was $V$, what is the value $V^star$ I should pay for the information?
How I would do - I would evaluate $H(X)$ using the definition, then I'd declare a new random variable $Y$ so that $mathcalA_Y = mathcalA_X ,/, x_0$ and $p_Y(y) = fracp_x(y)1 - p_x(x_0)$ and finally I'd evaluate $H(Y)$. Now I guess that $V^star = fracH(Y)H(X)V$.
I think this approach could be valid but I am not sure about that since it would be more natural to me to evaluate something like $H(X ,|, X neq x_0)$. Also, I'm not sure that a proportion is the way to go for finding $V^star$.
random-variables entropy
asked Jul 28 at 18:22
Giulio Scattolin
15619
Given a random variable $X$ having finite alphabet $mathcalA_X$ and valid $p_x(cdot)$ (for which there is no $x_0 in mathcalA_X$ so that $p_x(x_0) = 0$) I want to know its actual outcome (speaking about sport, $X$ could be the team that would buy a really famous player: the experts would give me $mathcalA_X$ and $p_x(cdot)$).
I'm willing to pay a $V$ amount of money for this information and I've found someone who could help me. Before the closing of the deal I find out that the outcome of $X$ won't surely be, let's say, $x_0$.
Since prior this discovery the value of $X$ was $V$, what is the value $V^star$ I should pay for the information?
How I would do - I would evaluate $H(X)$ using the definition, then I'd declare a new random variable $Y$ so that $mathcalA_Y = mathcalA_X ,/, x_0$ and $p_Y(y) = fracp_x(y)1 - p_x(x_0)$ and finally I'd evaluate $H(Y)$. Now I guess that $V^star = fracH(Y)H(X)V$.
I think this approach could be valid but I am not sure about that since it would be more natural to me to evaluate something like $H(X ,|, X neq x_0)$. Also, I'm not sure that a proportion is the way to go for finding $V^star$.
random-variables entropy
asked Jul 28 at 18:22
Giulio Scattolin
15619
edited Aug 4 at 13:11
asked Jul 28 at 18:22
Giulio Scattolin
15619
asked Jul 28 at 18:22
Giulio Scattolin
15619
asked Jul 28 at 18:22
Giulio Scattolin
15619
15619
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05
 |Â
show 2 more comments
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05
 |Â
show 2 more comments
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
Let's investigate if entropy can be a reasonable measure for this usage. Consider a rv $X$ whose alphabet is $mathcalA_X = alpha, beta, gamma $ and $p_X(alpha) = 0.9$, $p_X(beta) = 0.05$ and $p_X(gamma) = 0.05$. The entropy of $X$ defined as $$H(X) = -sum_x in mathcalA_X p_X(x) log_2(p_X(x))$$ gives $H(X) simeq 0.57$. For example, let's say we're sure $alpha$ won't be the outcome of $X$, we define $Y$ as stated in the question and evaluate its entropy obtaining $H(Y) = 1$. Considering another situation, this time $gamma$ won't be the outcome of $X$ and if we define $Z$ as said above we get $H(Z) simeq 0.3$.
In the initial scenario $alpha$ is the most likely outcome, almost certain. If we remove that outcome we're left with no more than a coin toss, the worst case scenario: our knowledge about $X$ is actually lower than before (we thought we were almost certain about the outcome - not entirely sure) and now we're left with a 50-50 decision. Knowing the exact outcome now is more valuable than knowing it before - this is represented by the fact that $H(Y) > H(X)$.
On the other hand, removing $gamma$ from the possible outcomes increases the probability that $alpha$ will be the real outcome, lowering the uncertainty of the scenario and the value of the information I wanted to buy - confirmed by $H(Z) < H(X)$.
This example seems to agree that entropy is a reasonable measure for this usage.
answered Aug 4 at 13:10
Giulio Scattolin
15619
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
Let's investigate if entropy can be a reasonable measure for this usage. Consider a rv $X$ whose alphabet is $mathcalA_X = alpha, beta, gamma $ and $p_X(alpha) = 0.9$, $p_X(beta) = 0.05$ and $p_X(gamma) = 0.05$. The entropy of $X$ defined as $$H(X) = -sum_x in mathcalA_X p_X(x) log_2(p_X(x))$$ gives $H(X) simeq 0.57$. For example, let's say we're sure $alpha$ won't be the outcome of $X$, we define $Y$ as stated in the question and evaluate its entropy obtaining $H(Y) = 1$. Considering another situation, this time $gamma$ won't be the outcome of $X$ and if we define $Z$ as said above we get $H(Z) simeq 0.3$.
In the initial scenario $alpha$ is the most likely outcome, almost certain. If we remove that outcome we're left with no more than a coin toss, the worst case scenario: our knowledge about $X$ is actually lower than before (we thought we were almost certain about the outcome - not entirely sure) and now we're left with a 50-50 decision. Knowing the exact outcome now is more valuable than knowing it before - this is represented by the fact that $H(Y) > H(X)$.
On the other hand, removing $gamma$ from the possible outcomes increases the probability that $alpha$ will be the real outcome, lowering the uncertainty of the scenario and the value of the information I wanted to buy - confirmed by $H(Z) < H(X)$.
This example seems to agree that entropy is a reasonable measure for this usage.
answered Aug 4 at 13:10
Giulio Scattolin
15619
add a comment |Â
up vote
0
down vote
accepted
Let's investigate if entropy can be a reasonable measure for this usage. Consider a rv $X$ whose alphabet is $mathcalA_X = alpha, beta, gamma $ and $p_X(alpha) = 0.9$, $p_X(beta) = 0.05$ and $p_X(gamma) = 0.05$. The entropy of $X$ defined as $$H(X) = -sum_x in mathcalA_X p_X(x) log_2(p_X(x))$$ gives $H(X) simeq 0.57$. For example, let's say we're sure $alpha$ won't be the outcome of $X$, we define $Y$ as stated in the question and evaluate its entropy obtaining $H(Y) = 1$. Considering another situation, this time $gamma$ won't be the outcome of $X$ and if we define $Z$ as said above we get $H(Z) simeq 0.3$.
In the initial scenario $alpha$ is the most likely outcome, almost certain. If we remove that outcome we're left with no more than a coin toss, the worst case scenario: our knowledge about $X$ is actually lower than before (we thought we were almost certain about the outcome - not entirely sure) and now we're left with a 50-50 decision. Knowing the exact outcome now is more valuable than knowing it before - this is represented by the fact that $H(Y) > H(X)$.
On the other hand, removing $gamma$ from the possible outcomes increases the probability that $alpha$ will be the real outcome, lowering the uncertainty of the scenario and the value of the information I wanted to buy - confirmed by $H(Z) < H(X)$.
This example seems to agree that entropy is a reasonable measure for this usage.
answered Aug 4 at 13:10
Giulio Scattolin
15619
add a comment |Â
up vote
0
down vote
accepted
up vote
0
down vote
accepted
Let's investigate if entropy can be a reasonable measure for this usage. Consider a rv $X$ whose alphabet is $mathcalA_X = alpha, beta, gamma $ and $p_X(alpha) = 0.9$, $p_X(beta) = 0.05$ and $p_X(gamma) = 0.05$. The entropy of $X$ defined as $$H(X) = -sum_x in mathcalA_X p_X(x) log_2(p_X(x))$$ gives $H(X) simeq 0.57$. For example, let's say we're sure $alpha$ won't be the outcome of $X$, we define $Y$ as stated in the question and evaluate its entropy obtaining $H(Y) = 1$. Considering another situation, this time $gamma$ won't be the outcome of $X$ and if we define $Z$ as said above we get $H(Z) simeq 0.3$.
In the initial scenario $alpha$ is the most likely outcome, almost certain. If we remove that outcome we're left with no more than a coin toss, the worst case scenario: our knowledge about $X$ is actually lower than before (we thought we were almost certain about the outcome - not entirely sure) and now we're left with a 50-50 decision. Knowing the exact outcome now is more valuable than knowing it before - this is represented by the fact that $H(Y) > H(X)$.
On the other hand, removing $gamma$ from the possible outcomes increases the probability that $alpha$ will be the real outcome, lowering the uncertainty of the scenario and the value of the information I wanted to buy - confirmed by $H(Z) < H(X)$.
This example seems to agree that entropy is a reasonable measure for this usage.
answered Aug 4 at 13:10
Giulio Scattolin
15619
Let's investigate if entropy can be a reasonable measure for this usage. Consider a rv $X$ whose alphabet is $mathcalA_X = alpha, beta, gamma $ and $p_X(alpha) = 0.9$, $p_X(beta) = 0.05$ and $p_X(gamma) = 0.05$. The entropy of $X$ defined as $$H(X) = -sum_x in mathcalA_X p_X(x) log_2(p_X(x))$$ gives $H(X) simeq 0.57$. For example, let's say we're sure $alpha$ won't be the outcome of $X$, we define $Y$ as stated in the question and evaluate its entropy obtaining $H(Y) = 1$. Considering another situation, this time $gamma$ won't be the outcome of $X$ and if we define $Z$ as said above we get $H(Z) simeq 0.3$.
In the initial scenario $alpha$ is the most likely outcome, almost certain. If we remove that outcome we're left with no more than a coin toss, the worst case scenario: our knowledge about $X$ is actually lower than before (we thought we were almost certain about the outcome - not entirely sure) and now we're left with a 50-50 decision. Knowing the exact outcome now is more valuable than knowing it before - this is represented by the fact that $H(Y) > H(X)$.
On the other hand, removing $gamma$ from the possible outcomes increases the probability that $alpha$ will be the real outcome, lowering the uncertainty of the scenario and the value of the information I wanted to buy - confirmed by $H(Z) < H(X)$.
This example seems to agree that entropy is a reasonable measure for this usage.
answered Aug 4 at 13:10
Giulio Scattolin
15619
answered Aug 4 at 13:10
Giulio Scattolin
15619
answered Aug 4 at 13:10
Giulio Scattolin
15619
answered Aug 4 at 13:10
Giulio Scattolin
15619
15619
add a comment |Â
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2865472%2fhow-to-evaluate-the-change-of-information-of-a-random-variable%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
It seems to me that you are actually computing "$H(X|Xneq x_0)$" with your approach (unless you mean something different by $H(X|Xneq x_0)$, which is an unconventional notation). Regarding $V^*$, this is essentially your own task to define how it is computed, however, the proportional approach seems certainly reasonable.
– Stelios
Jul 28 at 19:01
You can use Shannon's entropy law to calculate the information gain, but how much you should pay for the information is impossible to say.
– Björn Lindqvist
Jul 28 at 19:11
@Stelios thank you, I was unsure if $H(X|X leq x_0)$ was equivalent (somehow) to $H(Y)$: to my understanding the latter should be always less than the former since I thought it was reasonable to think that "knowing more" about a rv should lower its uncertanity (and entropy) but if $mathcalA_X = A,B,C$ and $p_X(A)=0.9$, $p_X(B)=0.05$, $p_X(C)=0.05$ then $H(X) simeq 0.57$. Removing $A$ from $X$ I get $Y$ and its entropy is $H(Y) = 1$, so $H(Y) > H(X)$. This means that the information I wanted to buy has "more value" now than before?
– Giulio Scattolin
Jul 28 at 19:27
@GiulioScattolin Your example is correct and serves to show that entropy is not an appropriate metric for your purposes. Note that entropy provides information about a random variable before it is observed. In particular, a high entropy random variable is less easy to predict. Of course, since in your case you want to increase your chances to correctly predict, a higher entropy variable is not useful. Maybe you should consider a different measure of information?
– Stelios
Jul 28 at 19:45
@Stelios I agree with that, can you suggest me what to look for? BTW I also tried to evaluate the entropy of $Z$ where $A_Z=A_X/C$ and it turns out that $H(Z)simeq 0.3<H(X)$. Actually this could make sense: the outcome is more predictable in the “ $Z$-situation †than the “ $X$ †one, so, if $V^starstar$ is what I’d pay in the “ $Z$-situation â€Â, then: $V^starstar < V < V^star$ (and $H(Z) < H(X) < H(Y)$). If that’s is correct then it is just a matter of choosing $mu : textentropymapstotextvalue$ so that $mu(H(X))=V$ and so on. Does it make any sense? Thank you very much.
– Giulio Scattolin
Jul 28 at 20:05