Variance of Monte Carlo integration with importance sampling





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







3












$begingroup$


I am following these lecture slides on Monte Carlo integration with importance sampling. I am just implementing a very simple example: $int_{0}^{1} e^{x}dx$. For the importance sampling version, I rewrite $int_{0}^{1} e^{x}dx = int_{0}^{1} e^{x}/p(x)cdot p(x)dx$ where $p(x) = 2.5x^{1.5}$. Then



$$hat{I} = frac{1}{N}sum_{j=1}^{N} frac{f(x_{j})}{p(x_{j})},$$



where $x_{j}$ are sampled from $p(x_{j})$ (I use an inverse transform method here). For the variance, I have $sigma_{I}^{2} = hat{sigma}_{I}^{2}/N$ and



$$hat{sigma}_{I}^{2} = frac{1}{N} sum_{j=1}^{N} frac{f(x_{j})^{2}}{g(x_{j})^{2}} - hat{I}^{2}.$$



I know I should expected the variance to decrease with importance sampling, but a plot of the variance with $N$ shows that not much happens. Can anyone explain to me what I'm doing incorrectly? I'm not sure how the they are able to achieve such a drastic decrease in variance in the lecture slides.



enter image description here






monte-carlo integral importance-sampling







share|cite|improve this question









$endgroup$



















    3












    $begingroup$


    I am following these lecture slides on Monte Carlo integration with importance sampling. I am just implementing a very simple example: $int_{0}^{1} e^{x}dx$. For the importance sampling version, I rewrite $int_{0}^{1} e^{x}dx = int_{0}^{1} e^{x}/p(x)cdot p(x)dx$ where $p(x) = 2.5x^{1.5}$. Then



    $$hat{I} = frac{1}{N}sum_{j=1}^{N} frac{f(x_{j})}{p(x_{j})},$$



    where $x_{j}$ are sampled from $p(x_{j})$ (I use an inverse transform method here). For the variance, I have $sigma_{I}^{2} = hat{sigma}_{I}^{2}/N$ and



    $$hat{sigma}_{I}^{2} = frac{1}{N} sum_{j=1}^{N} frac{f(x_{j})^{2}}{g(x_{j})^{2}} - hat{I}^{2}.$$



    I know I should expected the variance to decrease with importance sampling, but a plot of the variance with $N$ shows that not much happens. Can anyone explain to me what I'm doing incorrectly? I'm not sure how the they are able to achieve such a drastic decrease in variance in the lecture slides.



    enter image description here






    monte-carlo integral importance-sampling







    share|cite|improve this question









    $endgroup$















      3












      3








      3


      3



      $begingroup$


      I am following these lecture slides on Monte Carlo integration with importance sampling. I am just implementing a very simple example: $int_{0}^{1} e^{x}dx$. For the importance sampling version, I rewrite $int_{0}^{1} e^{x}dx = int_{0}^{1} e^{x}/p(x)cdot p(x)dx$ where $p(x) = 2.5x^{1.5}$. Then



      $$hat{I} = frac{1}{N}sum_{j=1}^{N} frac{f(x_{j})}{p(x_{j})},$$



      where $x_{j}$ are sampled from $p(x_{j})$ (I use an inverse transform method here). For the variance, I have $sigma_{I}^{2} = hat{sigma}_{I}^{2}/N$ and



      $$hat{sigma}_{I}^{2} = frac{1}{N} sum_{j=1}^{N} frac{f(x_{j})^{2}}{g(x_{j})^{2}} - hat{I}^{2}.$$



      I know I should expected the variance to decrease with importance sampling, but a plot of the variance with $N$ shows that not much happens. Can anyone explain to me what I'm doing incorrectly? I'm not sure how the they are able to achieve such a drastic decrease in variance in the lecture slides.



      enter image description here






      monte-carlo integral importance-sampling







      share|cite|improve this question









      $endgroup$




      I am following these lecture slides on Monte Carlo integration with importance sampling. I am just implementing a very simple example: $int_{0}^{1} e^{x}dx$. For the importance sampling version, I rewrite $int_{0}^{1} e^{x}dx = int_{0}^{1} e^{x}/p(x)cdot p(x)dx$ where $p(x) = 2.5x^{1.5}$. Then



      $$hat{I} = frac{1}{N}sum_{j=1}^{N} frac{f(x_{j})}{p(x_{j})},$$



      where $x_{j}$ are sampled from $p(x_{j})$ (I use an inverse transform method here). For the variance, I have $sigma_{I}^{2} = hat{sigma}_{I}^{2}/N$ and



      $$hat{sigma}_{I}^{2} = frac{1}{N} sum_{j=1}^{N} frac{f(x_{j})^{2}}{g(x_{j})^{2}} - hat{I}^{2}.$$



      I know I should expected the variance to decrease with importance sampling, but a plot of the variance with $N$ shows that not much happens. Can anyone explain to me what I'm doing incorrectly? I'm not sure how the they are able to achieve such a drastic decrease in variance in the lecture slides.



      enter image description here






      monte-carlo integral importance-sampling




      monte-carlo integral importance-sampling






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked Apr 1 at 18:26









      user1799323user1799323

      1234




      1234






















          1 Answer
          1






          active

          oldest

          votes


















          3












          $begingroup$

          enter image description hereThis is a good illustration of the dangers of importance sampling: while
          $$int_0^1 frac{e^x}{p(x)}, p(x)text{d} x = int_0^1 e^x text{d} x = I$$
          shows that $hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since
          $$int_0^1 left(frac{e^x}{p(x)}right)^2, p(x)text{d} x = int_0^1 frac{e^{2x}}{2.5 x^{1.5}} text{d} x = infty$$
          since the integral diverges in $x=0$. For instance,



          > x=runif(1e7)^{1/2.5}
          
> range(exp(x)/x^{1.5})
          
[1]     2.718282 83403.685972


          shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since



           > mean(exp(x)/x^{1.5})/2.5
          
 [1] 1.717576
          
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
          
 [1] 2.070953e-06


          but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)






          share|cite|improve this answer











          $endgroup$














            Your Answer





            StackExchange.ifUsing("editor", function () {
            return StackExchange.using("mathjaxEditing", function () {
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            });
            });
            }, "mathjax-editing");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "65"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400628%2fvariance-of-monte-carlo-integration-with-importance-sampling%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            3












            $begingroup$

            enter image description hereThis is a good illustration of the dangers of importance sampling: while
            $$int_0^1 frac{e^x}{p(x)}, p(x)text{d} x = int_0^1 e^x text{d} x = I$$
            shows that $hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since
            $$int_0^1 left(frac{e^x}{p(x)}right)^2, p(x)text{d} x = int_0^1 frac{e^{2x}}{2.5 x^{1.5}} text{d} x = infty$$
            since the integral diverges in $x=0$. For instance,



            > x=runif(1e7)^{1/2.5}
            
> range(exp(x)/x^{1.5})
            
[1]     2.718282 83403.685972


            shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since



             > mean(exp(x)/x^{1.5})/2.5
            
 [1] 1.717576
            
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
            
 [1] 2.070953e-06


            but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)






            share|cite|improve this answer











            $endgroup$


















              3












              $begingroup$

              enter image description hereThis is a good illustration of the dangers of importance sampling: while
              $$int_0^1 frac{e^x}{p(x)}, p(x)text{d} x = int_0^1 e^x text{d} x = I$$
              shows that $hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since
              $$int_0^1 left(frac{e^x}{p(x)}right)^2, p(x)text{d} x = int_0^1 frac{e^{2x}}{2.5 x^{1.5}} text{d} x = infty$$
              since the integral diverges in $x=0$. For instance,



              > x=runif(1e7)^{1/2.5}
              
> range(exp(x)/x^{1.5})
              
[1]     2.718282 83403.685972


              shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since



               > mean(exp(x)/x^{1.5})/2.5
              
 [1] 1.717576
              
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
              
 [1] 2.070953e-06


              but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)






              share|cite|improve this answer











              $endgroup$
















                3












                3








                3





                $begingroup$

                enter image description hereThis is a good illustration of the dangers of importance sampling: while
                $$int_0^1 frac{e^x}{p(x)}, p(x)text{d} x = int_0^1 e^x text{d} x = I$$
                shows that $hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since
                $$int_0^1 left(frac{e^x}{p(x)}right)^2, p(x)text{d} x = int_0^1 frac{e^{2x}}{2.5 x^{1.5}} text{d} x = infty$$
                since the integral diverges in $x=0$. For instance,



                > x=runif(1e7)^{1/2.5}
                
> range(exp(x)/x^{1.5})
                
[1]     2.718282 83403.685972


                shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since



                 > mean(exp(x)/x^{1.5})/2.5
                
 [1] 1.717576
                
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
                
 [1] 2.070953e-06


                but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)






                share|cite|improve this answer











                $endgroup$



                enter image description hereThis is a good illustration of the dangers of importance sampling: while
                $$int_0^1 frac{e^x}{p(x)}, p(x)text{d} x = int_0^1 e^x text{d} x = I$$
                shows that $hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since
                $$int_0^1 left(frac{e^x}{p(x)}right)^2, p(x)text{d} x = int_0^1 frac{e^{2x}}{2.5 x^{1.5}} text{d} x = infty$$
                since the integral diverges in $x=0$. For instance,



                > x=runif(1e7)^{1/2.5}
                
> range(exp(x)/x^{1.5})
                
[1]     2.718282 83403.685972


                shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since



                 > mean(exp(x)/x^{1.5})/2.5
                
 [1] 1.717576
                
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
                
 [1] 2.070953e-06


                but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)







                share|cite|improve this answer














                share|cite|improve this answer



                share|cite|improve this answer








                edited Apr 1 at 19:02

























                answered Apr 1 at 18:44









                Xi'anXi'an

                59.1k897365




                59.1k897365






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Cross Validated!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400628%2fvariance-of-monte-carlo-integration-with-importance-sampling%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    How did Captain America manage to do this?

                    Image
                    2 How does Captain America lift Thor's hammer? In Avengers: Age of Ultron , he is not worthy but how he is worthy in this film. plot-explanation marvel-cinematic-universe avengers-age-of-ultron avengers-endgame share | improve this question edited 19 mins ago A J ♦ 42.7k 16 229 246 asked 39 mins ago Andi AR Andi AR 325 1 4 14
                    Read more

                    迪纳利

                    Image
                    迪纳利峰 位于迪纳利国家公园和保留区的迪纳利峰（旧名麦金利峰） 最高点 海拔 20,237英尺（6,168米）   NGVD 29 . [1] [2] 相對高度 20,073英尺（6,118米） 列表 各洲最高峰列表 特独立山峰 各国最高点列表 美国最高点  排名第一 [3] 坐标 63°04′10″N 151°00′27″W  /  63.06944°N 151.00750°W  / 63.06944; -151.00750  ( Mount McKinley ) 坐标： 63°04′10″N 151°00′27″W  /  63.06944°N 151.00750°W  / 63.06944; -151.00750  ( Mount McKinley )   地理 位置   美國阿拉斯加州迪纳利自治市镇 迪纳利国家公园和保留区 山脈 阿拉斯加山脉 攀山 首次登頂 （ 英语 ： First ascent ） 1913年6月7日 哈德逊·史塔克 哈里·卡斯坦斯 沃尔特·哈珀 罗伯特·塔特姆 最簡路線 西坡路线（攀登冰川/雪） 迪纳利 （北阿萨巴斯卡语支、 英语： Denali ， / d ᵻ ˈ n ɑː l i / ，直译“高山”），1917年至2015年官方名称为 麦金利山 （另译作 麦金利峰 ， 英语： Mount McKinley ），位于阿拉斯加州东南部、阿拉斯加山脉中段，海拔6194米，是北美洲最高峰，也是美国的最高峰。它的俄语名称是 Большая Гора ， 罗马化：Bolshaya Gora ，意思是“大山”。 迪纳利山地区拥有变幻莫测的高山风、典型的北极植被以及野生动植物。这里大部分地区终年积雪，山间经常浓雾不断，雾气在皑皑白雪中缭绕弥漫时，几百米之外的景物便不可见。夏季，迪纳利山的青青山坡上鲜花盛开，紫色的杜鹃和精巧的铃状石南花随处可见。迪纳利山还是世界登山爱好者的汇集之地。每年5月到7月有数百人登山，只有一半不到的人能登顶成功。一次登山约花3个星期。 2015年8月31日，美國政府宣佈將山名由“麦金利山”改回原名“迪纳利山”。美國內政部發表聲
                    Read more

                    南乌拉尔铁路局

                    Image
                    南乌拉尔铁路局 Южно-Уральская железная дорога 成立 1934年1月4日 国家   蘇聯（1934年—1991年）   俄羅斯（1991年—） 总部 车里雅宾斯克 隶属 俄罗斯铁路股份公司 路局代码 80 铁路里程 4,934.9公里 奖项 网址 http://yuzd.rzd.ru/ 查 论 编 南乌拉尔铁路局 （俄语： Южно-Уральская железная дорога ， 羅馬化： Yuzhno-Uralskaya zheleznaya doroga ，简称 ЮУЖД ）是俄罗斯铁路股份公司属下的铁路管理局之一，总部位于车里雅宾斯克，负责管辖俄罗斯联邦乌拉尔南部地区的铁路系统，范围包括奥伦堡州、车里雅宾斯克州、库尔干州的大部分铁路，以及萨马拉州、斯维尔德洛夫斯克州、巴什科尔托斯坦共和国的少部分铁路，并有部分铁路延伸至或穿过哈萨克斯坦境内。南乌拉尔铁路局的营业里程为4,934.9公里，与伏尔加铁路局、古比雪夫铁路局、斯维尔德洛夫斯克铁路局、西西伯利亚铁路局相邻，并与南方的哈萨克斯坦铁路连接。 外部链接 （俄文） Южно-Уральская железная дорога — филиал ОАО РЖД （俄文） 南乌拉尔铁路局管辖路线图（一） （俄文） 南乌拉尔铁路局管辖路线图（二） 查 论 编 俄罗斯铁路（集团）股份公司 十月铁路局 · 莫斯科铁路局 · 北高加索铁路局 · 高尔基铁路局 · 古比雪夫铁路局 · 北方铁路局 · 伏尔加铁路局 · 东南铁路局 · 斯维尔德洛夫斯克铁路局 · 南乌拉尔铁路局 · 西西伯利亚铁路局 · 克拉斯诺亚尔斯克铁路局 · 东西伯利亚铁路局 · 后贝加尔铁路局 · 远东铁路局 · 加里宁格勒铁路局 This page is only for reference, If you need detailed information, please check here
                    Read more