Is stochastic gradient descent pseudo-stochastic?












4












$begingroup$


I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.



My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.






machine-learning neural-networks gradient-descent sgd







share|cite|improve this question











$endgroup$

















    4












    $begingroup$


    I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.



    My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.






    machine-learning neural-networks gradient-descent sgd







    share|cite|improve this question











    $endgroup$















      4












      4








      4





      $begingroup$


      I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.



      My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.






      machine-learning neural-networks gradient-descent sgd







      share|cite|improve this question











      $endgroup$




      I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.



      My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.






      machine-learning neural-networks gradient-descent sgd




      machine-learning neural-networks gradient-descent sgd






      share|cite|improve this question















      share|cite|improve this question













      share|cite|improve this question




      share|cite|improve this question








      edited yesterday









      Sycorax

      41.9k12109206




      41.9k12109206










      asked yesterday









      IamanonIamanon

      303




      303






















          1 Answer
          1






          active

          oldest

          votes


















          6












          $begingroup$

          Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.



          Consider a shuffled deck of cards. You look at the top card and see $mathsf{A}spadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsf{A}spadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf{2}color{red}{heartsuit}$ or $mathsf{J}clubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsf{A}spadesuit$ face-up somewhere else.



          In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsf{A}spadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.



          You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?






          share|cite|improve this answer











          $endgroup$













            Your Answer





            StackExchange.ifUsing("editor", function () {
            return StackExchange.using("mathjaxEditing", function () {
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            });
            });
            }, "mathjax-editing");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "65"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f398540%2fis-stochastic-gradient-descent-pseudo-stochastic%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            6












            $begingroup$

            Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.



            Consider a shuffled deck of cards. You look at the top card and see $mathsf{A}spadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsf{A}spadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf{2}color{red}{heartsuit}$ or $mathsf{J}clubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsf{A}spadesuit$ face-up somewhere else.



            In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsf{A}spadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.



            You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?






            share|cite|improve this answer











            $endgroup$


















              6












              $begingroup$

              Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.



              Consider a shuffled deck of cards. You look at the top card and see $mathsf{A}spadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsf{A}spadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf{2}color{red}{heartsuit}$ or $mathsf{J}clubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsf{A}spadesuit$ face-up somewhere else.



              In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsf{A}spadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.



              You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?






              share|cite|improve this answer











              $endgroup$
















                6












                6








                6





                $begingroup$

                Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.



                Consider a shuffled deck of cards. You look at the top card and see $mathsf{A}spadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsf{A}spadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf{2}color{red}{heartsuit}$ or $mathsf{J}clubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsf{A}spadesuit$ face-up somewhere else.



                In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsf{A}spadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.



                You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?






                share|cite|improve this answer











                $endgroup$



                Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.



                Consider a shuffled deck of cards. You look at the top card and see $mathsf{A}spadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsf{A}spadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf{2}color{red}{heartsuit}$ or $mathsf{J}clubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsf{A}spadesuit$ face-up somewhere else.



                In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsf{A}spadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.



                You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?







                share|cite|improve this answer














                share|cite|improve this answer



                share|cite|improve this answer








                edited yesterday

























                answered yesterday









                SycoraxSycorax

                41.9k12109206




                41.9k12109206






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Cross Validated!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f398540%2fis-stochastic-gradient-descent-pseudo-stochastic%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    數位音樂下載

                    Image
                    數位音樂下載 為非以「實體」方式來販賣音樂的相關產品，而使用數位格式如mp3、AAC等文件格式來進行銷售販賣的一种音乐产业形式。著名的數位音樂下載服務商有iTunes Store、RecoChoku、Google Play音乐和微软Zune等。数字音乐下载在二十一世纪初一度风靡全球， [1] [2] 使得传统唱片产业走向衰落，但其自身也在2010年代流媒体音乐服务如Apple Music、Spotify等的挤压下迅速失去市场份额。 [3] [4] 数字音乐下载有便于携带、传播迅速、制作简单成本低等优点，与流媒体音乐相比也能更好保障音乐人权益； [5] 但它同时很可能带来侵犯版权和滥用的情形， [6] 也给音乐产业带来了诸多不利影响。数字音乐下载为音乐人和乐迷提供了更加便捷直接的交流渠道，也一定程度上推动了网络音乐人的出现和独立音乐的迅速发展。 [7] 目录 1 历史 1.1 技术积淀（1987-2000） 1.2 出现与兴起（2000-2004） 1.3 唱片市场的颠覆者（2004-2007） 1.4 巅峰和衰落（2005-2013） 1.5 摇摇欲坠的现状与不可预知的明天（2014至今） 2 主要文件格式 2.1 PCM编码格式 2.1.1 WAV 2.1.2 APE 2.1.3 FLAC 2.2 苹果推出的无损文件格式 2.2.1 AIFF 2.2.2 ALAC 2.3 有损压缩格式 2.3.1 MP3 2.3.2 AAC 3 主要市场 3.1 iTunes Store 3.2 Google Play音乐商店 3.3 亚马逊音乐商店 3.4 部分其他音乐商店 4 影响与评价 4.1 对音乐产业的影响 4.2 对科技行业的影响 4.3 正面评价 4.4 负面评价 4.4.1 侵权问题 4.4.2 经营模式不明晰 4.4.3 不适应移动时代 4.5 文化现象 5 延伸阅读 6 参考资料 历史 苹果iTunes+iPod的软硬件搭配让数字音...
                    Read more

                    When can things happen in Etherscan, such as the picture below?

                    Image
                    2 I don't know why balance of the address is 0. when can this situation? etherscan balances share | improve this question asked yesterday Ru Hi Ru Hi 11 1 New contributor Ru Hi is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment  |  ...
                    Read more

                    格利澤436b

                    Image
                    格利泽436b 太陽系外行星 太陽系外行星列表 艺术家笔下的格利泽436b 母恆星 母恆星 Gliese 436 星座 獅子座 赤經 ( α ) 11 h  42 m  11.0941 s [1] 赤緯 ( δ ) +26° 42′ 23.652″ [1] 距離 33.4 ly (10.2 pc) 光譜類型 M2.5 V [1] 軌道參數 半长轴 ( a ) 0.0291±0.0004 [2] AU 軌道離心率 ( e ) 0.150±0.012 [2] 公轉週期 ( P ) 2.643904±0.000005 [3] d (0.00723849 y) 軌道傾角 ( i ) 85.8 +0.21 −0.25 [3] ° 角距 ( θ ) 2.794 mas 近星點時間 ( T 0 ) 2,451,551.716 ±0.01 JD 半振幅 ( K ) 18.68±0.8 m/s 物理性质 质量 ( m ) 22.2±1.0 [2] M ⊕ 半徑 ( r ) 4.327±0.183 [2] [4] R ⊕ 密度 ( ρ ) 1.51 g cm -3 表面重力 ( g ) 1.18 g 溫度 ( T ) 712±36 [2] K 發現 發現時間 2004 發現者 巴特勒、沃格特、 馬西 et al. 發現方法 徑向速度、凌日法 發現地點 美國加利福尼亞州 發表論文 出版 其他名稱 Ross 905 b, GJ 436 b, [5] LTT 13213 b, GCTP 2704.10 b, LHS 310, AC+27:28217 b, Vyssotsky 616 b, HIP 57087 b, GEN# +9.80120068 b, LP 319-75 b, G 121-7 b, LSPM J1142+2642 b, 1RXS J114211.9+264328 b, ASCC 683818 b, G 147-68 b, UCAC2 41198281 b, BP...
                    Read more