Under what conditions does the function C = f(A,B) satisfy H(C|A) = H(B)?
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above is true.
information-theory
asked 2 days ago
hklelhklel
1255
$endgroup$
add a comment |
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above is true.
information-theory
asked 2 days ago
hklelhklel
1255
$endgroup$
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago
add a comment |
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above is true.
information-theory
asked 2 days ago
hklelhklel
1255
$endgroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above is true.
information-theory
information-theory
asked 2 days ago
hklelhklel
1255
asked 2 days ago
hklelhklel
1255
asked 2 days ago
hklelhklel
1255
asked 2 days ago
hklelhklel
1255
asked 2 days ago
hklelhklel
1255
1255
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago
add a comment |
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago
1
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
answered 2 days ago
xskxzrxskxzr
4,05921033
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "419"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
add a comment |
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
add a comment |
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
$endgroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
edited 2 days ago
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
answered 2 days ago
Yuval FilmusYuval Filmus
195k14184349
195k14184349
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
add a comment |
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
1
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
2 days ago
3
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
answered 2 days ago
xskxzrxskxzr
4,05921033
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
answered 2 days ago
xskxzrxskxzr
4,05921033
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
answered 2 days ago
xskxzrxskxzr
4,05921033
$endgroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
answered 2 days ago
xskxzrxskxzr
4,05921033
edited 2 days ago
answered 2 days ago
xskxzrxskxzr
4,05921033
answered 2 days ago
xskxzrxskxzr
4,05921033
answered 2 days ago
xskxzrxskxzr
4,05921033
4,05921033
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
add a comment |
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
2
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
2 days ago
add a comment |
Thanks for contributing an answer to Computer Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
2 days ago
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
2 days ago