Deriving Probability Theory from Information Theory

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
0
down vote

favorite












In the paper "A Philosophical Treatise of Universal Induction" section 3 on Probability describes three different interpretations of probability theory: frequentist, objectivist, and subjectivist.



I am a proponent of concise definitions of fields. The most elegant definition I know of for Mathematics is: The study of patterns. I believe the best such definition for probability is: The study of patterns with incomplete information.



It's all about working with what we know to arrive at a best estimate of systems for which we have incomplete information.



In that light, it seems like one could derive probability theory from information theory. I know that historically, information theory is based on probability theory, but I wonder if one could reformulate information theory independent of probability theory, then derive probability theory from information theory.



Has that been attempted? Does it make any sense? Would it be of any value? I'm sorry if this is a malformed question. I'm not a mathematician by training. It simply seems like information is a more fundamental concept that probability should have been based upon but for historical reasons, the opposite happened.



EDIT: (For clarification)



Instead of defining information in terms of probability:



I(m) := -log(Pr(M=m)) // log base 2 for information in bits



It seems like one could define probability in terms of information (or lack there of):



Pr(M=m) := b^(-I(m)) // b=2 for information given in bits



If we take the definition of mathematics from above: The study of patterns
We could build a formal language for describing patterns from a set of symbols (say 0,1). Then we could use that language (as we have) to build pattern-based models that approximate the mechanics of the world (or imaginary worlds). Information could be defined as the length of those descriptions in units related to the alphabet (bits if the alphabet is 0,1). Then, when dealing with systems for which we have incomplete information, we can derive probability theory to help us infer properties of the system or make optimal decisions with the information at hand or incorporate new information into our model.



That's the basic idea. The benefit, as with other interpretations, is to gain insight from viewing problems from a different perspective. We currently use information indirectly, for instance; when assigning priors. It might allow us to develop a more rigorous approach in practice.




probability probability-theory definition information-theory kolmogorov-complexity




share|cite|improve this question

















  • 1




    See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
    – David G. Stork
    Jul 24 at 23:54






  • 2




    How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
    – Michael
    Jul 25 at 0:49











  • Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
    – arachnivore
    Jul 25 at 2:39










  • It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
    – arachnivore
    Jul 25 at 2:47










  • Thanks for the suggestion, David G. Stork!
    – arachnivore
    Jul 25 at 2:52














up vote
0
down vote

favorite












In the paper "A Philosophical Treatise of Universal Induction" section 3 on Probability describes three different interpretations of probability theory: frequentist, objectivist, and subjectivist.



I am a proponent of concise definitions of fields. The most elegant definition I know of for Mathematics is: The study of patterns. I believe the best such definition for probability is: The study of patterns with incomplete information.



It's all about working with what we know to arrive at a best estimate of systems for which we have incomplete information.



In that light, it seems like one could derive probability theory from information theory. I know that historically, information theory is based on probability theory, but I wonder if one could reformulate information theory independent of probability theory, then derive probability theory from information theory.



Has that been attempted? Does it make any sense? Would it be of any value? I'm sorry if this is a malformed question. I'm not a mathematician by training. It simply seems like information is a more fundamental concept that probability should have been based upon but for historical reasons, the opposite happened.



EDIT: (For clarification)



Instead of defining information in terms of probability:



I(m) := -log(Pr(M=m)) // log base 2 for information in bits



It seems like one could define probability in terms of information (or lack there of):



Pr(M=m) := b^(-I(m)) // b=2 for information given in bits



If we take the definition of mathematics from above: The study of patterns
We could build a formal language for describing patterns from a set of symbols (say 0,1). Then we could use that language (as we have) to build pattern-based models that approximate the mechanics of the world (or imaginary worlds). Information could be defined as the length of those descriptions in units related to the alphabet (bits if the alphabet is 0,1). Then, when dealing with systems for which we have incomplete information, we can derive probability theory to help us infer properties of the system or make optimal decisions with the information at hand or incorporate new information into our model.



That's the basic idea. The benefit, as with other interpretations, is to gain insight from viewing problems from a different perspective. We currently use information indirectly, for instance; when assigning priors. It might allow us to develop a more rigorous approach in practice.




probability probability-theory definition information-theory kolmogorov-complexity




share|cite|improve this question

















  • 1




    See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
    – David G. Stork
    Jul 24 at 23:54






  • 2




    How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
    – Michael
    Jul 25 at 0:49











  • Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
    – arachnivore
    Jul 25 at 2:39










  • It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
    – arachnivore
    Jul 25 at 2:47










  • Thanks for the suggestion, David G. Stork!
    – arachnivore
    Jul 25 at 2:52












up vote
0
down vote

favorite









up vote
0
down vote

favorite











In the paper "A Philosophical Treatise of Universal Induction" section 3 on Probability describes three different interpretations of probability theory: frequentist, objectivist, and subjectivist.



I am a proponent of concise definitions of fields. The most elegant definition I know of for Mathematics is: The study of patterns. I believe the best such definition for probability is: The study of patterns with incomplete information.



It's all about working with what we know to arrive at a best estimate of systems for which we have incomplete information.



In that light, it seems like one could derive probability theory from information theory. I know that historically, information theory is based on probability theory, but I wonder if one could reformulate information theory independent of probability theory, then derive probability theory from information theory.



Has that been attempted? Does it make any sense? Would it be of any value? I'm sorry if this is a malformed question. I'm not a mathematician by training. It simply seems like information is a more fundamental concept that probability should have been based upon but for historical reasons, the opposite happened.



EDIT: (For clarification)



Instead of defining information in terms of probability:



I(m) := -log(Pr(M=m)) // log base 2 for information in bits



It seems like one could define probability in terms of information (or lack there of):



Pr(M=m) := b^(-I(m)) // b=2 for information given in bits



If we take the definition of mathematics from above: The study of patterns
We could build a formal language for describing patterns from a set of symbols (say 0,1). Then we could use that language (as we have) to build pattern-based models that approximate the mechanics of the world (or imaginary worlds). Information could be defined as the length of those descriptions in units related to the alphabet (bits if the alphabet is 0,1). Then, when dealing with systems for which we have incomplete information, we can derive probability theory to help us infer properties of the system or make optimal decisions with the information at hand or incorporate new information into our model.



That's the basic idea. The benefit, as with other interpretations, is to gain insight from viewing problems from a different perspective. We currently use information indirectly, for instance; when assigning priors. It might allow us to develop a more rigorous approach in practice.




probability probability-theory definition information-theory kolmogorov-complexity




share|cite|improve this question













In the paper "A Philosophical Treatise of Universal Induction" section 3 on Probability describes three different interpretations of probability theory: frequentist, objectivist, and subjectivist.



I am a proponent of concise definitions of fields. The most elegant definition I know of for Mathematics is: The study of patterns. I believe the best such definition for probability is: The study of patterns with incomplete information.



It's all about working with what we know to arrive at a best estimate of systems for which we have incomplete information.



In that light, it seems like one could derive probability theory from information theory. I know that historically, information theory is based on probability theory, but I wonder if one could reformulate information theory independent of probability theory, then derive probability theory from information theory.



Has that been attempted? Does it make any sense? Would it be of any value? I'm sorry if this is a malformed question. I'm not a mathematician by training. It simply seems like information is a more fundamental concept that probability should have been based upon but for historical reasons, the opposite happened.



EDIT: (For clarification)



Instead of defining information in terms of probability:



I(m) := -log(Pr(M=m)) // log base 2 for information in bits



It seems like one could define probability in terms of information (or lack there of):



Pr(M=m) := b^(-I(m)) // b=2 for information given in bits



If we take the definition of mathematics from above: The study of patterns
We could build a formal language for describing patterns from a set of symbols (say 0,1). Then we could use that language (as we have) to build pattern-based models that approximate the mechanics of the world (or imaginary worlds). Information could be defined as the length of those descriptions in units related to the alphabet (bits if the alphabet is 0,1). Then, when dealing with systems for which we have incomplete information, we can derive probability theory to help us infer properties of the system or make optimal decisions with the information at hand or incorporate new information into our model.



That's the basic idea. The benefit, as with other interpretations, is to gain insight from viewing problems from a different perspective. We currently use information indirectly, for instance; when assigning priors. It might allow us to develop a more rigorous approach in practice.





probability probability-theory definition information-theory kolmogorov-complexity





share|cite|improve this question












share|cite|improve this question




share|cite|improve this question








edited Jul 25 at 19:24
























asked Jul 24 at 23:17









arachnivore

11




11







  • 1




    See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
    – David G. Stork
    Jul 24 at 23:54






  • 2




    How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
    – Michael
    Jul 25 at 0:49











  • Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
    – arachnivore
    Jul 25 at 2:39










  • It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
    – arachnivore
    Jul 25 at 2:47










  • Thanks for the suggestion, David G. Stork!
    – arachnivore
    Jul 25 at 2:52












  • 1




    See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
    – David G. Stork
    Jul 24 at 23:54






  • 2




    How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
    – Michael
    Jul 25 at 0:49











  • Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
    – arachnivore
    Jul 25 at 2:39










  • It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
    – arachnivore
    Jul 25 at 2:47










  • Thanks for the suggestion, David G. Stork!
    – arachnivore
    Jul 25 at 2:52







1




1




See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
– David G. Stork
Jul 24 at 23:54




See David MacKay's book: amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/…
– David G. Stork
Jul 24 at 23:54




2




2




How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
– Michael
Jul 25 at 0:49





How do you propose to derive probability theory from information theory? And why? The concepts of entropy and mutual information require probability. It is like deriving arithmetic from calculus.
– Michael
Jul 25 at 0:49













Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
– arachnivore
Jul 25 at 2:39




Well, again I'm no mathematician, but the idea is that the probability we estimate for some event is contingent upon the information we have of the relevant factors. Maybe instead of defining information in terms of probability: I(m) = -log(p(m)) It makes more sense to define probability in terms of information: p(m) = 2^-I(m) Paul Revere may have estimated the probability of the Brits arriving by sea as 50% but the Brits knew it was 100% and had Revere factored in information that was relevant to the decision, he could have arrived at a better estimate.
– arachnivore
Jul 25 at 2:39












It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
– arachnivore
Jul 25 at 2:47




It seems like one could start with something like a formal language constructed from a finite set of symbols (say 0,1) and define information in terms of numbers of symbols, then bring in Turing and Kolmogorov etc. and work up to a probability-free theory of information. Then define probability in terms of information instead of vice-versa. It's all about dealing with systems for which we have incomplete information. It makes more sense to me than a frequentist, subjective, or objective interpretation.
– arachnivore
Jul 25 at 2:47












Thanks for the suggestion, David G. Stork!
– arachnivore
Jul 25 at 2:52




Thanks for the suggestion, David G. Stork!
– arachnivore
Jul 25 at 2:52















active

oldest

votes











Your Answer




StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "69"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2861854%2fderiving-probability-theory-from-information-theory%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes










 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2861854%2fderiving-probability-theory-from-information-theory%23new-answer', 'question_page');

);

Post as a guest
































































Comments

Post a Comment

Popular posts from this blog

What is the equation of a 3D cone with generalised tilt?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite What is the equation of a 3D cone with generalized tilt? I've noticed that in most equations given to represent a cone, there is no parameter which defines the tilt of the cone in 3D space and that most of them have their point at the origin $(0,0,0)$ - I was wondering if anyone could give me a more generalized cone equation for a cone in any position in 3D space 3d share | cite | improve this question edited Jul 25 '16 at 0:05 ervx 9,425 3 13 36 asked Jul 24 '16 at 15:38 Charlene 11 2 2 Welcome to Math.SE! Could you please explain a few things to help people give an answer useful to you: 1. What do you mean by "generalized tilt"? 2. Are you asking about a right circular cone? Does the angle at the vertex matter? 3. What form of answer (implicit, parametric...?) are you looking for? 4. If this is homework, what tools do you have available, an...
Read more

Color the edges and diagonals of a regular polygon

Image
Clash Royale CLAN TAG #URR8PPP up vote 5 down vote favorite 1 Here is the problem: For what $n$ is it possible to color the edges and diagonals of an $n$-side regular polygon with $dfracbinomn23$ colors, such that you use every color exactly three times and for every color the three segments (edges or diagonals) with that color form a triangle? Trivially $nequiv 0 text or 1 pmod3$. I can also prove that if the statement is true for $k$ than it is true for $3k$ as well. How to finish? Please help! Thanks combinatorial-geometry share | cite | improve this question edited Aug 6 at 21:57 alcana 118 4 asked Aug 6 at 21:15 Leo Gardner 356 11 Even assuming "polyhpn" in the title is a typo for polygon , it is unclear what you mean by "the edges and sides". Aren't the side of a regular polygon the same as the edges? A regular $n$-sided polygon has $n$ edges. – hardmath Aug 6 at 21:20 3 $n$ ne...
Read more

Relationship between determinant of matrix and determinant of adjoint?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite We are studying adjoints in class, and I was curious if there is a relationship between the determinant of matrix A, and the determinant of the adjoint of matrix A? I assume there would be a relationship because finding the adjoint requires creating a cofactor matrix and then transposing it. linear-algebra determinant adjoint-operators share | cite | improve this question asked Nov 17 '17 at 17:32 CluelessCoder 32 5 For a square matrix $A$ of order $n$, we have $A(operatornameadj A)=(operatornameadj A)A=|A|I_n$ where $I_n$ is the identity matrix of order $n$. This should tell you about the determinant of the adjoint in terms of that of $A$ (use multiplicative property and other elementary properties of determinants). – Prasun Biswas Nov 17 '17 at 17:35 1 @CluelessCoder: see here: math.stackexchange.com/questions/516127/…...
Read more