Backpropagation: Derivaiton of Loss Gradient

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite
1












In the book "Artifical Intelligence: A Modern Approach" from S. Russel there is a derivation of the gradient of the loss with respect to the weights w used for backpropagation on page 735.



I stumbled across the third row of the derivation, where $Delta_k$ is inserted.
On the left side of the equation, there is still a '2', but on the right side it is gone.



Can someone explain to me what happened to that '2'?



begineqnarray
fracpartial Loss_kpartial omega_j,k &=& -2(y_k - a_k)fracpartial a_kpartial omega_j,k = -2(y_k - a_k)fracpartial g(in_k)partial omega_j,k \
&=& -2(y_k - a_k)g'(in_k)fracpartial in_kpartial omega_j,k = -2(y_k - a_k)g'(in_k)fracpartial partial omega_j,kleft( sum _j omega_j,ka_jright) \
&=& -colorred2(y_k - a_k) g'(in_k) a_j = -a_jDelta_k
endeqnarray



where



$$
Delta_k = Err_k times g'(in_k)
$$




vector-analysis machine-learning gradient-descent neural-networks




share|cite|improve this question





















  • Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
    – José Carlos Santos
    Jul 16 at 12:41










  • How is the quantity $Err_k$ defined?
    – caverac
    Jul 16 at 14:32










  • 'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
    – MJA
    Jul 16 at 15:49















up vote
1
down vote

favorite
1












In the book "Artifical Intelligence: A Modern Approach" from S. Russel there is a derivation of the gradient of the loss with respect to the weights w used for backpropagation on page 735.



I stumbled across the third row of the derivation, where $Delta_k$ is inserted.
On the left side of the equation, there is still a '2', but on the right side it is gone.



Can someone explain to me what happened to that '2'?



begineqnarray
fracpartial Loss_kpartial omega_j,k &=& -2(y_k - a_k)fracpartial a_kpartial omega_j,k = -2(y_k - a_k)fracpartial g(in_k)partial omega_j,k \
&=& -2(y_k - a_k)g'(in_k)fracpartial in_kpartial omega_j,k = -2(y_k - a_k)g'(in_k)fracpartial partial omega_j,kleft( sum _j omega_j,ka_jright) \
&=& -colorred2(y_k - a_k) g'(in_k) a_j = -a_jDelta_k
endeqnarray



where



$$
Delta_k = Err_k times g'(in_k)
$$




vector-analysis machine-learning gradient-descent neural-networks




share|cite|improve this question





















  • Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
    – José Carlos Santos
    Jul 16 at 12:41










  • How is the quantity $Err_k$ defined?
    – caverac
    Jul 16 at 14:32










  • 'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
    – MJA
    Jul 16 at 15:49













up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





In the book "Artifical Intelligence: A Modern Approach" from S. Russel there is a derivation of the gradient of the loss with respect to the weights w used for backpropagation on page 735.



I stumbled across the third row of the derivation, where $Delta_k$ is inserted.
On the left side of the equation, there is still a '2', but on the right side it is gone.



Can someone explain to me what happened to that '2'?



begineqnarray
fracpartial Loss_kpartial omega_j,k &=& -2(y_k - a_k)fracpartial a_kpartial omega_j,k = -2(y_k - a_k)fracpartial g(in_k)partial omega_j,k \
&=& -2(y_k - a_k)g'(in_k)fracpartial in_kpartial omega_j,k = -2(y_k - a_k)g'(in_k)fracpartial partial omega_j,kleft( sum _j omega_j,ka_jright) \
&=& -colorred2(y_k - a_k) g'(in_k) a_j = -a_jDelta_k
endeqnarray



where



$$
Delta_k = Err_k times g'(in_k)
$$




vector-analysis machine-learning gradient-descent neural-networks




share|cite|improve this question













In the book "Artifical Intelligence: A Modern Approach" from S. Russel there is a derivation of the gradient of the loss with respect to the weights w used for backpropagation on page 735.



I stumbled across the third row of the derivation, where $Delta_k$ is inserted.
On the left side of the equation, there is still a '2', but on the right side it is gone.



Can someone explain to me what happened to that '2'?



begineqnarray
fracpartial Loss_kpartial omega_j,k &=& -2(y_k - a_k)fracpartial a_kpartial omega_j,k = -2(y_k - a_k)fracpartial g(in_k)partial omega_j,k \
&=& -2(y_k - a_k)g'(in_k)fracpartial in_kpartial omega_j,k = -2(y_k - a_k)g'(in_k)fracpartial partial omega_j,kleft( sum _j omega_j,ka_jright) \
&=& -colorred2(y_k - a_k) g'(in_k) a_j = -a_jDelta_k
endeqnarray



where



$$
Delta_k = Err_k times g'(in_k)
$$





vector-analysis machine-learning gradient-descent neural-networks





share|cite|improve this question












share|cite|improve this question




share|cite|improve this question








edited Jul 16 at 14:31









caverac

11k2927




11k2927









asked Jul 16 at 12:38









MJA

62




62











  • Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
    – José Carlos Santos
    Jul 16 at 12:41










  • How is the quantity $Err_k$ defined?
    – caverac
    Jul 16 at 14:32










  • 'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
    – MJA
    Jul 16 at 15:49

















  • Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
    – José Carlos Santos
    Jul 16 at 12:41










  • How is the quantity $Err_k$ defined?
    – caverac
    Jul 16 at 14:32










  • 'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
    – MJA
    Jul 16 at 15:49
















Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
– José Carlos Santos
Jul 16 at 12:41




Welcome to MSE. It is in your best interest that you type your questions (using MathJax) instead of posting links to pictures.
– José Carlos Santos
Jul 16 at 12:41












How is the quantity $Err_k$ defined?
– caverac
Jul 16 at 14:32




How is the quantity $Err_k$ defined?
– caverac
Jul 16 at 14:32












'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
– MJA
Jul 16 at 15:49





'Errk be the kth component of the error vector y − hw': So Errk is the same as (yk - ak)
– MJA
Jul 16 at 15:49
















active

oldest

votes











Your Answer




StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "69"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2853367%2fbackpropagation-derivaiton-of-loss-gradient%23new-answer', 'question_page');

);

Post as a guest






















active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes










 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2853367%2fbackpropagation-derivaiton-of-loss-gradient%23new-answer', 'question_page');

);

Post as a guest
































































Comments

Post a Comment

Popular posts from this blog

What is the equation of a 3D cone with generalised tilt?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite What is the equation of a 3D cone with generalized tilt? I've noticed that in most equations given to represent a cone, there is no parameter which defines the tilt of the cone in 3D space and that most of them have their point at the origin $(0,0,0)$ - I was wondering if anyone could give me a more generalized cone equation for a cone in any position in 3D space 3d share | cite | improve this question edited Jul 25 '16 at 0:05 ervx 9,425 3 13 36 asked Jul 24 '16 at 15:38 Charlene 11 2 2 Welcome to Math.SE! Could you please explain a few things to help people give an answer useful to you: 1. What do you mean by "generalized tilt"? 2. Are you asking about a right circular cone? Does the angle at the vertex matter? 3. What form of answer (implicit, parametric...?) are you looking for? 4. If this is homework, what tools do you have available, an...
Read more

Color the edges and diagonals of a regular polygon

Image
Clash Royale CLAN TAG #URR8PPP up vote 5 down vote favorite 1 Here is the problem: For what $n$ is it possible to color the edges and diagonals of an $n$-side regular polygon with $dfracbinomn23$ colors, such that you use every color exactly three times and for every color the three segments (edges or diagonals) with that color form a triangle? Trivially $nequiv 0 text or 1 pmod3$. I can also prove that if the statement is true for $k$ than it is true for $3k$ as well. How to finish? Please help! Thanks combinatorial-geometry share | cite | improve this question edited Aug 6 at 21:57 alcana 118 4 asked Aug 6 at 21:15 Leo Gardner 356 11 Even assuming "polyhpn" in the title is a typo for polygon , it is unclear what you mean by "the edges and sides". Aren't the side of a regular polygon the same as the edges? A regular $n$-sided polygon has $n$ edges. – hardmath Aug 6 at 21:20 3 $n$ ne...
Read more

Relationship between determinant of matrix and determinant of adjoint?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite We are studying adjoints in class, and I was curious if there is a relationship between the determinant of matrix A, and the determinant of the adjoint of matrix A? I assume there would be a relationship because finding the adjoint requires creating a cofactor matrix and then transposing it. linear-algebra determinant adjoint-operators share | cite | improve this question asked Nov 17 '17 at 17:32 CluelessCoder 32 5 For a square matrix $A$ of order $n$, we have $A(operatornameadj A)=(operatornameadj A)A=|A|I_n$ where $I_n$ is the identity matrix of order $n$. This should tell you about the determinant of the adjoint in terms of that of $A$ (use multiplicative property and other elementary properties of determinants). – Prasun Biswas Nov 17 '17 at 17:35 1 @CluelessCoder: see here: math.stackexchange.com/questions/516127/…...
Read more