Mixing
entropy of the sum of binomial distributions
Clash Royale CLAN TAG#URR8PPP
up vote
2
down vote
favorite
Suppose that one has $X_1 sim Bin(n,p)$ and $X_2 sim Bin(n,1-p)$ and that $Z$ is distributed s.t:
$$
P(Z = k) = .5 P(X_1=k) + .5P(X_2=k)
$$
How do we compute the entropy? For a binomial distributed variable i have seen this proof:
Entropy of a binomial distribution
let $A = p^k (1-p)^n-k + (1-p)^k (1-p)^n-k$ then we see that:
beginalign
H(Z) &= -.5 sum n choose k Big[ABig] log Big[.5n choose kABig] \&= 1 - .5sum n choose k Big[ABig] log Big[n choose kBig] - .5sum n choose k Big[ABig] log Big[ABig]
endalign
Using de-Moivre-Laplace theorem, i get:
beginalign*
1+log_2(sqrt2pisigma) + int_-infty^infty Big[e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2)Big] log_2(e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2))
endalign*
where $mu_1 = np, mu2 = n(1-p)$ and $sigma^2 = np(1-p)$
I tried to get the integral on the right hand side using Mathematica but that failed. Any suggestions on how to get the exact or approximation of this integral? What i tried is this:
beginalign*
R &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) \ &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]Big[log_2left(e^frac(x-mu_1)^22sigma^2right)+log_2left(1+e^frac-(2p-1)(n-2x)2(p-1)pright)Big]
\& approx frac12sigmasqrt2 piint_-infty^inftyBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(left(e^frac(x-mu_1)^22sigma^2right)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p \&= frac14log_2(e)left(1 + (sigma^2 + (n-2mu_1)^2)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p
endalign*
The approximation
$log_2(1+e^frac-(2p-1)(n-2x)2(p-1)p) approx e^frac-(2p-1)(n-2x)2(p-1)p textif x in [fracn2,infty)$ and since the function is symetrical around $fracn2$ i can just multiply the integration by 2. The right hand side gives me an ugly term so i was wondering if an approximation there could work?
Are there more theorems that i should consider looking at?
binomial-coefficients binomial-distribution entropy
asked Jul 25 at 14:59
Kees Til
577617
add a comment |Â
up vote
2
down vote
favorite
Suppose that one has $X_1 sim Bin(n,p)$ and $X_2 sim Bin(n,1-p)$ and that $Z$ is distributed s.t:
$$
P(Z = k) = .5 P(X_1=k) + .5P(X_2=k)
$$
How do we compute the entropy? For a binomial distributed variable i have seen this proof:
Entropy of a binomial distribution
let $A = p^k (1-p)^n-k + (1-p)^k (1-p)^n-k$ then we see that:
beginalign
H(Z) &= -.5 sum n choose k Big[ABig] log Big[.5n choose kABig] \&= 1 - .5sum n choose k Big[ABig] log Big[n choose kBig] - .5sum n choose k Big[ABig] log Big[ABig]
endalign
Using de-Moivre-Laplace theorem, i get:
beginalign*
1+log_2(sqrt2pisigma) + int_-infty^infty Big[e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2)Big] log_2(e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2))
endalign*
where $mu_1 = np, mu2 = n(1-p)$ and $sigma^2 = np(1-p)$
I tried to get the integral on the right hand side using Mathematica but that failed. Any suggestions on how to get the exact or approximation of this integral? What i tried is this:
beginalign*
R &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) \ &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]Big[log_2left(e^frac(x-mu_1)^22sigma^2right)+log_2left(1+e^frac-(2p-1)(n-2x)2(p-1)pright)Big]
\& approx frac12sigmasqrt2 piint_-infty^inftyBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(left(e^frac(x-mu_1)^22sigma^2right)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p \&= frac14log_2(e)left(1 + (sigma^2 + (n-2mu_1)^2)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p
endalign*
The approximation
$log_2(1+e^frac-(2p-1)(n-2x)2(p-1)p) approx e^frac-(2p-1)(n-2x)2(p-1)p textif x in [fracn2,infty)$ and since the function is symetrical around $fracn2$ i can just multiply the integration by 2. The right hand side gives me an ugly term so i was wondering if an approximation there could work?
Are there more theorems that i should consider looking at?
binomial-coefficients binomial-distribution entropy
asked Jul 25 at 14:59
Kees Til
577617
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
1
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57
add a comment |Â
up vote
2
down vote
favorite
up vote
2
down vote
favorite
Suppose that one has $X_1 sim Bin(n,p)$ and $X_2 sim Bin(n,1-p)$ and that $Z$ is distributed s.t:
$$
P(Z = k) = .5 P(X_1=k) + .5P(X_2=k)
$$
How do we compute the entropy? For a binomial distributed variable i have seen this proof:
Entropy of a binomial distribution
let $A = p^k (1-p)^n-k + (1-p)^k (1-p)^n-k$ then we see that:
beginalign
H(Z) &= -.5 sum n choose k Big[ABig] log Big[.5n choose kABig] \&= 1 - .5sum n choose k Big[ABig] log Big[n choose kBig] - .5sum n choose k Big[ABig] log Big[ABig]
endalign
Using de-Moivre-Laplace theorem, i get:
beginalign*
1+log_2(sqrt2pisigma) + int_-infty^infty Big[e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2)Big] log_2(e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2))
endalign*
where $mu_1 = np, mu2 = n(1-p)$ and $sigma^2 = np(1-p)$
I tried to get the integral on the right hand side using Mathematica but that failed. Any suggestions on how to get the exact or approximation of this integral? What i tried is this:
beginalign*
R &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) \ &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]Big[log_2left(e^frac(x-mu_1)^22sigma^2right)+log_2left(1+e^frac-(2p-1)(n-2x)2(p-1)pright)Big]
\& approx frac12sigmasqrt2 piint_-infty^inftyBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(left(e^frac(x-mu_1)^22sigma^2right)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p \&= frac14log_2(e)left(1 + (sigma^2 + (n-2mu_1)^2)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p
endalign*
The approximation
$log_2(1+e^frac-(2p-1)(n-2x)2(p-1)p) approx e^frac-(2p-1)(n-2x)2(p-1)p textif x in [fracn2,infty)$ and since the function is symetrical around $fracn2$ i can just multiply the integration by 2. The right hand side gives me an ugly term so i was wondering if an approximation there could work?
Are there more theorems that i should consider looking at?
binomial-coefficients binomial-distribution entropy
asked Jul 25 at 14:59
Kees Til
577617
Suppose that one has $X_1 sim Bin(n,p)$ and $X_2 sim Bin(n,1-p)$ and that $Z$ is distributed s.t:
$$
P(Z = k) = .5 P(X_1=k) + .5P(X_2=k)
$$
How do we compute the entropy? For a binomial distributed variable i have seen this proof:
Entropy of a binomial distribution
let $A = p^k (1-p)^n-k + (1-p)^k (1-p)^n-k$ then we see that:
beginalign
H(Z) &= -.5 sum n choose k Big[ABig] log Big[.5n choose kABig] \&= 1 - .5sum n choose k Big[ABig] log Big[n choose kBig] - .5sum n choose k Big[ABig] log Big[ABig]
endalign
Using de-Moivre-Laplace theorem, i get:
beginalign*
1+log_2(sqrt2pisigma) + int_-infty^infty Big[e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2)Big] log_2(e^frac(x-mu_1)^22sigma^2)+e^frac(x-mu_2)^22sigma^2))
endalign*
where $mu_1 = np, mu2 = n(1-p)$ and $sigma^2 = np(1-p)$
I tried to get the integral on the right hand side using Mathematica but that failed. Any suggestions on how to get the exact or approximation of this integral? What i tried is this:
beginalign*
R &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) \ &= int_-infty^inftyfrac12sigmasqrt2 piBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]Big[log_2left(e^frac(x-mu_1)^22sigma^2right)+log_2left(1+e^frac-(2p-1)(n-2x)2(p-1)pright)Big]
\& approx frac12sigmasqrt2 piint_-infty^inftyBig[e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2Big]log_2left(left(e^frac(x-mu_1)^22sigma^2right)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p \&= frac14log_2(e)left(1 + (sigma^2 + (n-2mu_1)^2)right) + frac1sigmasqrt2 pi int_fracn2^inftyleft(e^-frac(x-mu_1)^22sigma^2+e^-frac(x-mu_2)^22sigma^2right) e^frac-(2p-1)(n-2x)2(p-1)p
endalign*
The approximation
$log_2(1+e^frac-(2p-1)(n-2x)2(p-1)p) approx e^frac-(2p-1)(n-2x)2(p-1)p textif x in [fracn2,infty)$ and since the function is symetrical around $fracn2$ i can just multiply the integration by 2. The right hand side gives me an ugly term so i was wondering if an approximation there could work?
Are there more theorems that i should consider looking at?
binomial-coefficients binomial-distribution entropy
asked Jul 25 at 14:59
Kees Til
577617
edited Jul 31 at 13:24
asked Jul 25 at 14:59
Kees Til
577617
asked Jul 25 at 14:59
Kees Til
577617
asked Jul 25 at 14:59
Kees Til
577617
577617
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
1
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57
add a comment |Â
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
1
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
1
1
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
2
down vote
For small $n$, you just compute it numerically.
If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:
$$H(B;n,p)=frac1 2 log_2 big( 2pi e, np(1-p) big) + O left( frac1n right) tag1$$
You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $alpha,1-alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get
$$H(Z)=H(A)+H(Zmid A) - H(Amid Z)=h(alpha)+alpha H(X) + (1-alpha)H(Y)- H(Amid Z) tag2$$
where $h(cdot)$ is the binary entropy function. In our case, $alpha=frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence
$$H(Z)=1 + H(B;n,p) - H(A mid Z) tag3$$
Because $0le H(A mid Z)le1$,
$$H(B;n,p)le H(Z)le 1 + H(B;n,p) $$
Assuming $0<p<1$ and $nto infty$, then $$ H(Z) =
frac1 2 log_2 big( 2pi e, np(1-p) big) + O(1)\
approx frac1 2 log_2 big( 2pi e, np(1-p) big)$$
answered Jul 25 at 15:21
leonbloy
38k644104
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
For small $n$, you just compute it numerically.
If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:
$$H(B;n,p)=frac1 2 log_2 big( 2pi e, np(1-p) big) + O left( frac1n right) tag1$$
You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $alpha,1-alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get
$$H(Z)=H(A)+H(Zmid A) - H(Amid Z)=h(alpha)+alpha H(X) + (1-alpha)H(Y)- H(Amid Z) tag2$$
where $h(cdot)$ is the binary entropy function. In our case, $alpha=frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence
$$H(Z)=1 + H(B;n,p) - H(A mid Z) tag3$$
Because $0le H(A mid Z)le1$,
$$H(B;n,p)le H(Z)le 1 + H(B;n,p) $$
Assuming $0<p<1$ and $nto infty$, then $$ H(Z) =
frac1 2 log_2 big( 2pi e, np(1-p) big) + O(1)\
approx frac1 2 log_2 big( 2pi e, np(1-p) big)$$
answered Jul 25 at 15:21
leonbloy
38k644104
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
add a comment |Â
up vote
2
down vote
For small $n$, you just compute it numerically.
If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:
$$H(B;n,p)=frac1 2 log_2 big( 2pi e, np(1-p) big) + O left( frac1n right) tag1$$
You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $alpha,1-alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get
$$H(Z)=H(A)+H(Zmid A) - H(Amid Z)=h(alpha)+alpha H(X) + (1-alpha)H(Y)- H(Amid Z) tag2$$
where $h(cdot)$ is the binary entropy function. In our case, $alpha=frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence
$$H(Z)=1 + H(B;n,p) - H(A mid Z) tag3$$
Because $0le H(A mid Z)le1$,
$$H(B;n,p)le H(Z)le 1 + H(B;n,p) $$
Assuming $0<p<1$ and $nto infty$, then $$ H(Z) =
frac1 2 log_2 big( 2pi e, np(1-p) big) + O(1)\
approx frac1 2 log_2 big( 2pi e, np(1-p) big)$$
answered Jul 25 at 15:21
leonbloy
38k644104
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
add a comment |Â
up vote
2
down vote
up vote
2
down vote
For small $n$, you just compute it numerically.
If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:
$$H(B;n,p)=frac1 2 log_2 big( 2pi e, np(1-p) big) + O left( frac1n right) tag1$$
You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $alpha,1-alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get
$$H(Z)=H(A)+H(Zmid A) - H(Amid Z)=h(alpha)+alpha H(X) + (1-alpha)H(Y)- H(Amid Z) tag2$$
where $h(cdot)$ is the binary entropy function. In our case, $alpha=frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence
$$H(Z)=1 + H(B;n,p) - H(A mid Z) tag3$$
Because $0le H(A mid Z)le1$,
$$H(B;n,p)le H(Z)le 1 + H(B;n,p) $$
Assuming $0<p<1$ and $nto infty$, then $$ H(Z) =
frac1 2 log_2 big( 2pi e, np(1-p) big) + O(1)\
approx frac1 2 log_2 big( 2pi e, np(1-p) big)$$
answered Jul 25 at 15:21
leonbloy
38k644104
For small $n$, you just compute it numerically.
If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:
$$H(B;n,p)=frac1 2 log_2 big( 2pi e, np(1-p) big) + O left( frac1n right) tag1$$
You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $alpha,1-alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get
$$H(Z)=H(A)+H(Zmid A) - H(Amid Z)=h(alpha)+alpha H(X) + (1-alpha)H(Y)- H(Amid Z) tag2$$
where $h(cdot)$ is the binary entropy function. In our case, $alpha=frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence
$$H(Z)=1 + H(B;n,p) - H(A mid Z) tag3$$
Because $0le H(A mid Z)le1$,
$$H(B;n,p)le H(Z)le 1 + H(B;n,p) $$
Assuming $0<p<1$ and $nto infty$, then $$ H(Z) =
frac1 2 log_2 big( 2pi e, np(1-p) big) + O(1)\
approx frac1 2 log_2 big( 2pi e, np(1-p) big)$$
answered Jul 25 at 15:21
leonbloy
38k644104
edited Jul 28 at 21:40
answered Jul 25 at 15:21
leonbloy
38k644104
answered Jul 25 at 15:21
leonbloy
38k644104
answered Jul 25 at 15:21
leonbloy
38k644104
38k644104
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
add a comment |Â
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
I really like the way of thinking in this proof, do you think for small $n$ only numerical approximations are possible?
– Kees Til
Jul 25 at 15:31
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
For small $n$, one can hardly think of something better than just computing it numerically - even the entropy of a single Binomial
– leonbloy
Jul 25 at 15:45
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2862497%2fentropy-of-the-sum-of-binomial-distributions%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
The entropy will vary between $H_0$ and $H_0+1$, where $H_0$ is the entropy of a single binomial. (For $p=0.5$ you're just reproducing the binomial, and for $pto0$ you have one extra bit of information for which branch was used.) So you could try to subtract out the $H_0$ part and approximate the rest as a function between $0$ and $1$.
– joriki
Jul 25 at 15:14
How did you come up with these bounds? I do understand it has an upper bound of 1 and lower bound of 0, is that what you mean?
– Kees Til
Jul 25 at 15:18
What are you referring to by "it"?
– joriki
Jul 25 at 15:25
the entropy but you clarified everything in your answer thanks :D
– Kees Til
Jul 25 at 15:36
1
I didn't write an answer; it's by leonbloy.
– joriki
Jul 25 at 15:57