How do we know the LHC results are robust?












58












$begingroup$


Nature article on reproducibility in science.



According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.



Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results. In fact, because 50% of physics & engineering experiments aren't reproducible by the original scientists, one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?



EDIT: As pointed out by Chris Hayes in the comments, I misinterpreted the Nature article. It says that 50% of physical scientists have failed to reproduce their own results, which is not the same statement as 50% of physics experiments aren't reproducible. This significantly eases the concern I had when I wrote the question. I'm leaving the question here however, because the core idea - how can we know the LHC's results are robust when we only have one LHC? - remains the same, and because innisfree wrote an excellent answer.










share|cite|improve this question











$endgroup$








  • 10




    $begingroup$
    I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
    $endgroup$
    – Aaron Stevens
    2 days ago






  • 5




    $begingroup$
    I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
    $endgroup$
    – Aaron Stevens
    2 days ago








  • 7




    $begingroup$
    For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
    $endgroup$
    – rob
    2 days ago






  • 21




    $begingroup$
    @Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
    $endgroup$
    – Chris Hayes
    2 days ago






  • 3




    $begingroup$
    @ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
    $endgroup$
    – jkej
    2 days ago
















58












$begingroup$


Nature article on reproducibility in science.



According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.



Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results. In fact, because 50% of physics & engineering experiments aren't reproducible by the original scientists, one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?



EDIT: As pointed out by Chris Hayes in the comments, I misinterpreted the Nature article. It says that 50% of physical scientists have failed to reproduce their own results, which is not the same statement as 50% of physics experiments aren't reproducible. This significantly eases the concern I had when I wrote the question. I'm leaving the question here however, because the core idea - how can we know the LHC's results are robust when we only have one LHC? - remains the same, and because innisfree wrote an excellent answer.










share|cite|improve this question











$endgroup$








  • 10




    $begingroup$
    I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
    $endgroup$
    – Aaron Stevens
    2 days ago






  • 5




    $begingroup$
    I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
    $endgroup$
    – Aaron Stevens
    2 days ago








  • 7




    $begingroup$
    For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
    $endgroup$
    – rob
    2 days ago






  • 21




    $begingroup$
    @Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
    $endgroup$
    – Chris Hayes
    2 days ago






  • 3




    $begingroup$
    @ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
    $endgroup$
    – jkej
    2 days ago














58












58








58


8



$begingroup$


Nature article on reproducibility in science.



According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.



Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results. In fact, because 50% of physics & engineering experiments aren't reproducible by the original scientists, one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?



EDIT: As pointed out by Chris Hayes in the comments, I misinterpreted the Nature article. It says that 50% of physical scientists have failed to reproduce their own results, which is not the same statement as 50% of physics experiments aren't reproducible. This significantly eases the concern I had when I wrote the question. I'm leaving the question here however, because the core idea - how can we know the LHC's results are robust when we only have one LHC? - remains the same, and because innisfree wrote an excellent answer.










share|cite|improve this question











$endgroup$




Nature article on reproducibility in science.



According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have been failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.



Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results. In fact, because 50% of physics & engineering experiments aren't reproducible by the original scientists, one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?



EDIT: As pointed out by Chris Hayes in the comments, I misinterpreted the Nature article. It says that 50% of physical scientists have failed to reproduce their own results, which is not the same statement as 50% of physics experiments aren't reproducible. This significantly eases the concern I had when I wrote the question. I'm leaving the question here however, because the core idea - how can we know the LHC's results are robust when we only have one LHC? - remains the same, and because innisfree wrote an excellent answer.







particle-physics large-hadron-collider data-analysis






share|cite|improve this question















share|cite|improve this question













share|cite|improve this question




share|cite|improve this question








edited 2 days ago







Allure

















asked 2 days ago









AllureAllure

2,272925




2,272925








  • 10




    $begingroup$
    I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
    $endgroup$
    – Aaron Stevens
    2 days ago






  • 5




    $begingroup$
    I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
    $endgroup$
    – Aaron Stevens
    2 days ago








  • 7




    $begingroup$
    For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
    $endgroup$
    – rob
    2 days ago






  • 21




    $begingroup$
    @Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
    $endgroup$
    – Chris Hayes
    2 days ago






  • 3




    $begingroup$
    @ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
    $endgroup$
    – jkej
    2 days ago














  • 10




    $begingroup$
    I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
    $endgroup$
    – Aaron Stevens
    2 days ago






  • 5




    $begingroup$
    I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
    $endgroup$
    – Aaron Stevens
    2 days ago








  • 7




    $begingroup$
    For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
    $endgroup$
    – rob
    2 days ago






  • 21




    $begingroup$
    @Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
    $endgroup$
    – Chris Hayes
    2 days ago






  • 3




    $begingroup$
    @ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
    $endgroup$
    – jkej
    2 days ago








10




10




$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
2 days ago




$begingroup$
I think it is worth pointing out that the LHC doesn't just do one particle collision and then say the experiment is completed. How much do you know about what goes into such experiments, how many times they are actually repeated, and then how the data is analyzed from there?
$endgroup$
– Aaron Stevens
2 days ago




5




5




$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
2 days ago






$begingroup$
I was asking if you had looked into the efforts taken to make sure the results from the LHC are good results and not just mistakes. Also the LHC isn't the only particle collider in existence.
$endgroup$
– Aaron Stevens
2 days ago






7




7




$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob
2 days ago




$begingroup$
For more about this important question than you may have bargained for, look for discussions about the "look-elsewhere effect" in the statistical analysis of the data from the mostly-independent ATLAS and CMS experiments at the LHC, especially in the context of their joint discovery of the Higgs particle.
$endgroup$
– rob
2 days ago




21




21




$begingroup$
@Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
$endgroup$
– Chris Hayes
2 days ago




$begingroup$
@Allure "Half [of scientists] have failed to reproduce their own experiments" is an enormously different statement from "half of all experiments are non-reproducible". The former statement (from the Nature article) includes scientists who have failed to reproduce a single one of their experiments, even if they successfully reproduced 99 out of 100. See page 10 of the questionnaire for the exact wording.
$endgroup$
– Chris Hayes
2 days ago




3




3




$begingroup$
@ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
$endgroup$
– jkej
2 days ago




$begingroup$
@ChrisHayes My thought exactly! Furthermore, a failed attempt to replicate an experiment doesn't necessarily mean that the original experiment is "non-reproducible".
$endgroup$
– jkej
2 days ago










4 Answers
4






active

oldest

votes


















82












$begingroup$

That's a really great question. The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including




  • Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect

  • Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.

  • Poor training in statistical methods.


I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:




  • Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$). To put that into perspective, the corresponding type-1 error rates are $0.05$ for $2sigma$ and about $3times10^{-7}$ for $5sigma$

  • The results from the LHC are already replicated!


    • There are several detectors placed around the LHC ring. Two of them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).

    • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.

    • The Higgs boson was observed in several datasets collected at the LHC



  • The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.

  • The LHC teams are guided by statistical committees, hopefully ensuring good practice

  • The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.

  • Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).

  • All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.

  • All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal

  • LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking

  • LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

  • The Higgs boson (or similar new physics) was theoretically required due to a 'no-lose' theorem about the breakdown of models without a Higgs at LHC energies, so we can be even more confident that it is a genuine effect.
    The other new effects that are being searched for at the LHC, however, arguably aren't as well motivated, so this doesn't apply to them. E.g., there was no a priori motivation for a 750 GeV resonanace that was hinted at in data but ultimately disappeared.


If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease. In this paper it was argued this was the case for SUSY searches in run-1.






share|cite|improve this answer











$endgroup$









  • 20




    $begingroup$
    This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
    $endgroup$
    – knzhou
    2 days ago








  • 3




    $begingroup$
    I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
    $endgroup$
    – mbrig
    2 days ago






  • 5




    $begingroup$
    Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
    $endgroup$
    – Rococo
    2 days ago












  • $begingroup$
    A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
    $endgroup$
    – Derek Elkins
    yesterday










  • $begingroup$
    all great points! thanks
    $endgroup$
    – innisfree
    yesterday



















12












$begingroup$

In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.



In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.



Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).






share|cite|improve this answer








New contributor




chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






$endgroup$









  • 1




    $begingroup$
    I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
    $endgroup$
    – Scrontch
    2 days ago










  • $begingroup$
    @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
    $endgroup$
    – chrylis
    2 days ago



















6












$begingroup$

The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.



Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?



What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?



The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.



from the question:



"because 50% of physics & engineering experiments aren't reproducible by the original scientists",



This is a fake statement from a dubious poll. The statistical significance of the "not reproducible " has not been checked in the poll. Only if it were a one standard deviation result , there exists almost a 50% a probability of the next trial not to reproduce.




one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results




No way, because engineering and physics analysis at the LHC are over the 4 sigma level, and the probability of negation is small. Even a 3sigma level has confidence 99% , so the chance is in no way 50%.



We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.



(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)



The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.



Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.






share|cite|improve this answer











$endgroup$









  • 11




    $begingroup$
    I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
    $endgroup$
    – Aaron Stevens
    2 days ago












  • $begingroup$
    @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
    $endgroup$
    – anna v
    2 days ago










  • $begingroup$
    Comments are not for extended discussion; this conversation has been moved to chat.
    $endgroup$
    – ACuriousMind
    40 mins ago



















4












$begingroup$

Any one experiment is repeated many times on the same equipment. They look for rare events, and it takes a lot of rare events to be sure that they aren't just coincidence.



The question about how many LHCs it takes to be sure, is different.



Each LHC component had to be carefully tested to make sure it was in spec. Remember the example of the experiment that seemed to get a result slightly faster than light. Because it was so important, they went to great expense to test everything, components around the world, until they found two components that were out of spec, that created the small error. If the error had been in the other direction would they have done that testing? No. They wouldn't even notice the error. It wouldn't be important. What made this one important was faster than light. Did they carefully record every out-of-spec component they found that would tend to slow the signal, that might cancel the positive errors they found? Maybe. That wasn't what they were looking for, though. That was a complication and not a solution to the problem.



After the tested LHC components are installed they must be tested again in case they were changed while being handled.



Then they must be calibrated. Every analog output could have a baseline that's a little bit off, because of random things. A solder joint that's slightly different. An AC circuit nearby that changes things a little bit every 120th of a second. The baseline must be calibrated for every one of them. Once the signal has been converted to digital then it's OK. Errors smaller than the cutoff are ignored, and larger errors make one bit difference. For the calibration, you know what the outcome is supposed to be, so you set it to that.



Could all this have somehow changed the outcomes so that some extremely unlikely results are falsely reported more often than they should be?



There's no theoretical reason to expect it. And the engineers who assembled the LHC were very very careful. But how could we test it? The obvious way is to build at least 2 more LHCs and notice how consistent their results are. That would be very expensive. It will not be done.



We can get some confidence by looking at results from other machinery. It's like -- the LHC was used to scan for a wide range of possible results that could be called the Higgs boson. They could do in years what a lesser machine might take centuries to do. But once we have a specific Higgs boson to look for, some of the others can look for that specifically and see whether they find it. If they do, then there's probably something there beyond equipment error.



Something else they can do (which I think they are doing part of the time) is look for things that are supposed to not happen that nobody predicts will happen. When they find one for sure then everybody will get excited. People will say there's something wrong, and insist that they check for every possible error that could give them that result. Like with the faster-than-light thing.






share|cite|improve this answer









$endgroup$














    Your Answer





    StackExchange.ifUsing("editor", function () {
    return StackExchange.using("mathjaxEditing", function () {
    StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
    StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
    });
    });
    }, "mathjax-editing");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "151"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    noCode: true, onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fphysics.stackexchange.com%2fquestions%2f468891%2fhow-do-we-know-the-lhc-results-are-robust%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    4 Answers
    4






    active

    oldest

    votes








    4 Answers
    4






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    82












    $begingroup$

    That's a really great question. The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including




    • Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect

    • Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.

    • Poor training in statistical methods.


    I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:




    • Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$). To put that into perspective, the corresponding type-1 error rates are $0.05$ for $2sigma$ and about $3times10^{-7}$ for $5sigma$

    • The results from the LHC are already replicated!


      • There are several detectors placed around the LHC ring. Two of them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).

      • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.

      • The Higgs boson was observed in several datasets collected at the LHC



    • The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.

    • The LHC teams are guided by statistical committees, hopefully ensuring good practice

    • The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.

    • Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).

    • All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.

    • All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal

    • LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking

    • LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

    • The Higgs boson (or similar new physics) was theoretically required due to a 'no-lose' theorem about the breakdown of models without a Higgs at LHC energies, so we can be even more confident that it is a genuine effect.
      The other new effects that are being searched for at the LHC, however, arguably aren't as well motivated, so this doesn't apply to them. E.g., there was no a priori motivation for a 750 GeV resonanace that was hinted at in data but ultimately disappeared.


    If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease. In this paper it was argued this was the case for SUSY searches in run-1.






    share|cite|improve this answer











    $endgroup$









    • 20




      $begingroup$
      This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
      $endgroup$
      – knzhou
      2 days ago








    • 3




      $begingroup$
      I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
      $endgroup$
      – mbrig
      2 days ago






    • 5




      $begingroup$
      Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
      $endgroup$
      – Rococo
      2 days ago












    • $begingroup$
      A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
      $endgroup$
      – Derek Elkins
      yesterday










    • $begingroup$
      all great points! thanks
      $endgroup$
      – innisfree
      yesterday
















    82












    $begingroup$

    That's a really great question. The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including




    • Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect

    • Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.

    • Poor training in statistical methods.


    I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:




    • Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$). To put that into perspective, the corresponding type-1 error rates are $0.05$ for $2sigma$ and about $3times10^{-7}$ for $5sigma$

    • The results from the LHC are already replicated!


      • There are several detectors placed around the LHC ring. Two of them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).

      • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.

      • The Higgs boson was observed in several datasets collected at the LHC



    • The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.

    • The LHC teams are guided by statistical committees, hopefully ensuring good practice

    • The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.

    • Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).

    • All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.

    • All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal

    • LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking

    • LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

    • The Higgs boson (or similar new physics) was theoretically required due to a 'no-lose' theorem about the breakdown of models without a Higgs at LHC energies, so we can be even more confident that it is a genuine effect.
      The other new effects that are being searched for at the LHC, however, arguably aren't as well motivated, so this doesn't apply to them. E.g., there was no a priori motivation for a 750 GeV resonanace that was hinted at in data but ultimately disappeared.


    If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease. In this paper it was argued this was the case for SUSY searches in run-1.






    share|cite|improve this answer











    $endgroup$









    • 20




      $begingroup$
      This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
      $endgroup$
      – knzhou
      2 days ago








    • 3




      $begingroup$
      I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
      $endgroup$
      – mbrig
      2 days ago






    • 5




      $begingroup$
      Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
      $endgroup$
      – Rococo
      2 days ago












    • $begingroup$
      A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
      $endgroup$
      – Derek Elkins
      yesterday










    • $begingroup$
      all great points! thanks
      $endgroup$
      – innisfree
      yesterday














    82












    82








    82





    $begingroup$

    That's a really great question. The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including




    • Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect

    • Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.

    • Poor training in statistical methods.


    I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:




    • Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$). To put that into perspective, the corresponding type-1 error rates are $0.05$ for $2sigma$ and about $3times10^{-7}$ for $5sigma$

    • The results from the LHC are already replicated!


      • There are several detectors placed around the LHC ring. Two of them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).

      • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.

      • The Higgs boson was observed in several datasets collected at the LHC



    • The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.

    • The LHC teams are guided by statistical committees, hopefully ensuring good practice

    • The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.

    • Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).

    • All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.

    • All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal

    • LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking

    • LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

    • The Higgs boson (or similar new physics) was theoretically required due to a 'no-lose' theorem about the breakdown of models without a Higgs at LHC energies, so we can be even more confident that it is a genuine effect.
      The other new effects that are being searched for at the LHC, however, arguably aren't as well motivated, so this doesn't apply to them. E.g., there was no a priori motivation for a 750 GeV resonanace that was hinted at in data but ultimately disappeared.


    If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease. In this paper it was argued this was the case for SUSY searches in run-1.






    share|cite|improve this answer











    $endgroup$



    That's a really great question. The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including




    • Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect

    • Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.

    • Poor training in statistical methods.


    I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:




    • Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$). To put that into perspective, the corresponding type-1 error rates are $0.05$ for $2sigma$ and about $3times10^{-7}$ for $5sigma$

    • The results from the LHC are already replicated!


      • There are several detectors placed around the LHC ring. Two of them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).

      • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.

      • The Higgs boson was observed in several datasets collected at the LHC



    • The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.

    • The LHC teams are guided by statistical committees, hopefully ensuring good practice

    • The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.

    • Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).

    • All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.

    • All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal

    • LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking

    • LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

    • The Higgs boson (or similar new physics) was theoretically required due to a 'no-lose' theorem about the breakdown of models without a Higgs at LHC energies, so we can be even more confident that it is a genuine effect.
      The other new effects that are being searched for at the LHC, however, arguably aren't as well motivated, so this doesn't apply to them. E.g., there was no a priori motivation for a 750 GeV resonanace that was hinted at in data but ultimately disappeared.


    If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease. In this paper it was argued this was the case for SUSY searches in run-1.







    share|cite|improve this answer














    share|cite|improve this answer



    share|cite|improve this answer








    edited yesterday

























    answered 2 days ago









    innisfreeinnisfree

    11.9k33162




    11.9k33162








    • 20




      $begingroup$
      This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
      $endgroup$
      – knzhou
      2 days ago








    • 3




      $begingroup$
      I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
      $endgroup$
      – mbrig
      2 days ago






    • 5




      $begingroup$
      Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
      $endgroup$
      – Rococo
      2 days ago












    • $begingroup$
      A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
      $endgroup$
      – Derek Elkins
      yesterday










    • $begingroup$
      all great points! thanks
      $endgroup$
      – innisfree
      yesterday














    • 20




      $begingroup$
      This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
      $endgroup$
      – knzhou
      2 days ago








    • 3




      $begingroup$
      I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
      $endgroup$
      – mbrig
      2 days ago






    • 5




      $begingroup$
      Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
      $endgroup$
      – Rococo
      2 days ago












    • $begingroup$
      A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
      $endgroup$
      – Derek Elkins
      yesterday










    • $begingroup$
      all great points! thanks
      $endgroup$
      – innisfree
      yesterday








    20




    20




    $begingroup$
    This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
    $endgroup$
    – knzhou
    2 days ago






    $begingroup$
    This is an excellent answer! I think it should be further emphasized how different $2sigma$ is from $5 sigma$. Using the standard $2 sigma$ conventions of social science, you have a 5% chance of getting a significant result every time you test a completely false hypothesis. (And this can easily be boosted by a factor of 10 by $p$-hacking techniques, plus you can report something like $p = 0.1$ as "trending towards significance".) Asking for $5 sigma$ is not merely being $5/2$ as rigorous, the corresponding $p$-value cutoff is roughly $0.0000003$.
    $endgroup$
    – knzhou
    2 days ago






    3




    3




    $begingroup$
    I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
    $endgroup$
    – mbrig
    2 days ago




    $begingroup$
    I think saying the "social sciences" is possibly a bit overly specific. There's been much talk and news of reproducibility problems in biology and chemistry as well, at least in semi-recent years, though perhaps not as bad as the social sciences are experiencing.
    $endgroup$
    – mbrig
    2 days ago




    5




    5




    $begingroup$
    Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
    $endgroup$
    – Rococo
    2 days ago






    $begingroup$
    Although it is a slightly different issue than the statistical considerations that are the focus of this answer, laymen also often don't appreciate that the LHC has necessarily reproduced many previous discoveries: atlas.cern/updates/atlas-blog/art-rediscovery . From these and similar studies, we can directly evaluate whether the reproducibility crisis seems to be present in particle physics... and, not surprisingly given the extensive measures described in your answer, it appears that it does better than many (all?) other fields, so far.
    $endgroup$
    – Rococo
    2 days ago














    $begingroup$
    A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
    $endgroup$
    – Derek Elkins
    yesterday




    $begingroup$
    A minor addition is that the prior probability for the Higgs boson existing and to a much lesser extent being in the range it was found is likely higher than "surprising" results in social science. Which is just to say it wasn't a surprise that the Higgs boson existed; that's what the theory predicted. Some new non-Higgs particle would warrant much more skepticism.
    $endgroup$
    – Derek Elkins
    yesterday












    $begingroup$
    all great points! thanks
    $endgroup$
    – innisfree
    yesterday




    $begingroup$
    all great points! thanks
    $endgroup$
    – innisfree
    yesterday











    12












    $begingroup$

    In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.



    In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.



    Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).






    share|cite|improve this answer








    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






    $endgroup$









    • 1




      $begingroup$
      I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
      $endgroup$
      – Scrontch
      2 days ago










    • $begingroup$
      @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
      $endgroup$
      – chrylis
      2 days ago
















    12












    $begingroup$

    In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.



    In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.



    Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).






    share|cite|improve this answer








    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






    $endgroup$









    • 1




      $begingroup$
      I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
      $endgroup$
      – Scrontch
      2 days ago










    • $begingroup$
      @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
      $endgroup$
      – chrylis
      2 days ago














    12












    12








    12





    $begingroup$

    In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.



    In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.



    Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).






    share|cite|improve this answer








    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






    $endgroup$



    In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.



    In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.



    Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).







    share|cite|improve this answer








    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.









    share|cite|improve this answer



    share|cite|improve this answer






    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.









    answered 2 days ago









    chrylischrylis

    22114




    22114




    New contributor




    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.





    New contributor





    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






    chrylis is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.








    • 1




      $begingroup$
      I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
      $endgroup$
      – Scrontch
      2 days ago










    • $begingroup$
      @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
      $endgroup$
      – chrylis
      2 days ago














    • 1




      $begingroup$
      I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
      $endgroup$
      – Scrontch
      2 days ago










    • $begingroup$
      @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
      $endgroup$
      – chrylis
      2 days ago








    1




    1




    $begingroup$
    I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
    $endgroup$
    – Scrontch
    2 days ago




    $begingroup$
    I understand what you mean, but "match experimental data with models" sounds like there is ample reason to expect confirmation bias if not done properly.
    $endgroup$
    – Scrontch
    2 days ago












    $begingroup$
    @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
    $endgroup$
    – chrylis
    2 days ago




    $begingroup$
    @Scrontch If not done properly, of course, but the useful property of these two questions (yes/no and value-in-range) is that we can run simulations ahead of time and define with clarity what the results should look like in the different possible universes, including such information as how wide the margins need to be to give us confidence. There are (fairly) standard ways of doing this.
    $endgroup$
    – chrylis
    2 days ago











    6












    $begingroup$

    The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.



    Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?



    What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?



    The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.



    from the question:



    "because 50% of physics & engineering experiments aren't reproducible by the original scientists",



    This is a fake statement from a dubious poll. The statistical significance of the "not reproducible " has not been checked in the poll. Only if it were a one standard deviation result , there exists almost a 50% a probability of the next trial not to reproduce.




    one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results




    No way, because engineering and physics analysis at the LHC are over the 4 sigma level, and the probability of negation is small. Even a 3sigma level has confidence 99% , so the chance is in no way 50%.



    We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.



    (And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)



    The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.



    Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.






    share|cite|improve this answer











    $endgroup$









    • 11




      $begingroup$
      I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
      $endgroup$
      – Aaron Stevens
      2 days ago












    • $begingroup$
      @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
      $endgroup$
      – anna v
      2 days ago










    • $begingroup$
      Comments are not for extended discussion; this conversation has been moved to chat.
      $endgroup$
      – ACuriousMind
      40 mins ago
















    6












    $begingroup$

    The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.



    Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?



    What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?



    The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.



    from the question:



    "because 50% of physics & engineering experiments aren't reproducible by the original scientists",



    This is a fake statement from a dubious poll. The statistical significance of the "not reproducible " has not been checked in the poll. Only if it were a one standard deviation result , there exists almost a 50% a probability of the next trial not to reproduce.




    one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results




    No way, because engineering and physics analysis at the LHC are over the 4 sigma level, and the probability of negation is small. Even a 3sigma level has confidence 99% , so the chance is in no way 50%.



    We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.



    (And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)



    The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.



    Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.






    share|cite|improve this answer











    $endgroup$









    • 11




      $begingroup$
      I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
      $endgroup$
      – Aaron Stevens
      2 days ago












    • $begingroup$
      @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
      $endgroup$
      – anna v
      2 days ago










    • $begingroup$
      Comments are not for extended discussion; this conversation has been moved to chat.
      $endgroup$
      – ACuriousMind
      40 mins ago














    6












    6








    6





    $begingroup$

    The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.



    Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?



    What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?



    The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.



    from the question:



    "because 50% of physics & engineering experiments aren't reproducible by the original scientists",



    This is a fake statement from a dubious poll. The statistical significance of the "not reproducible " has not been checked in the poll. Only if it were a one standard deviation result , there exists almost a 50% a probability of the next trial not to reproduce.




    one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results




    No way, because engineering and physics analysis at the LHC are over the 4 sigma level, and the probability of negation is small. Even a 3sigma level has confidence 99% , so the chance is in no way 50%.



    We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.



    (And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)



    The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.



    Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.






    share|cite|improve this answer











    $endgroup$



    The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.



    Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?



    What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?



    The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.



    from the question:



    "because 50% of physics & engineering experiments aren't reproducible by the original scientists",



    This is a fake statement from a dubious poll. The statistical significance of the "not reproducible " has not been checked in the poll. Only if it were a one standard deviation result , there exists almost a 50% a probability of the next trial not to reproduce.




    one might expect there's a 50% chance that if the people who originally built the LHC built another LHC, they will not reach the same results




    No way, because engineering and physics analysis at the LHC are over the 4 sigma level, and the probability of negation is small. Even a 3sigma level has confidence 99% , so the chance is in no way 50%.



    We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.



    (And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)



    The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.



    Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.







    share|cite|improve this answer














    share|cite|improve this answer



    share|cite|improve this answer








    edited 2 days ago

























    answered 2 days ago









    anna vanna v

    161k8153452




    161k8153452








    • 11




      $begingroup$
      I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
      $endgroup$
      – Aaron Stevens
      2 days ago












    • $begingroup$
      @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
      $endgroup$
      – anna v
      2 days ago










    • $begingroup$
      Comments are not for extended discussion; this conversation has been moved to chat.
      $endgroup$
      – ACuriousMind
      40 mins ago














    • 11




      $begingroup$
      I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
      $endgroup$
      – Aaron Stevens
      2 days ago












    • $begingroup$
      @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
      $endgroup$
      – anna v
      2 days ago










    • $begingroup$
      Comments are not for extended discussion; this conversation has been moved to chat.
      $endgroup$
      – ACuriousMind
      40 mins ago








    11




    11




    $begingroup$
    I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
    $endgroup$
    – Aaron Stevens
    2 days ago






    $begingroup$
    I'm not sure I buy the Olympics analogy. Failed attempts at breaking a record isn't the same thing as a failed attempt to reproduce an experiment. It also sounds like you are saying we should just cherry pick what does work and ignore when it fails.
    $endgroup$
    – Aaron Stevens
    2 days ago














    $begingroup$
    @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
    $endgroup$
    – anna v
    2 days ago




    $begingroup$
    @AaronStevens " cherry pick what does work" but is not that evolution in general? and "ignore when it fails" one learns from failure to design better experiments.
    $endgroup$
    – anna v
    2 days ago












    $begingroup$
    Comments are not for extended discussion; this conversation has been moved to chat.
    $endgroup$
    – ACuriousMind
    40 mins ago




    $begingroup$
    Comments are not for extended discussion; this conversation has been moved to chat.
    $endgroup$
    – ACuriousMind
    40 mins ago











    4












    $begingroup$

    Any one experiment is repeated many times on the same equipment. They look for rare events, and it takes a lot of rare events to be sure that they aren't just coincidence.



    The question about how many LHCs it takes to be sure, is different.



    Each LHC component had to be carefully tested to make sure it was in spec. Remember the example of the experiment that seemed to get a result slightly faster than light. Because it was so important, they went to great expense to test everything, components around the world, until they found two components that were out of spec, that created the small error. If the error had been in the other direction would they have done that testing? No. They wouldn't even notice the error. It wouldn't be important. What made this one important was faster than light. Did they carefully record every out-of-spec component they found that would tend to slow the signal, that might cancel the positive errors they found? Maybe. That wasn't what they were looking for, though. That was a complication and not a solution to the problem.



    After the tested LHC components are installed they must be tested again in case they were changed while being handled.



    Then they must be calibrated. Every analog output could have a baseline that's a little bit off, because of random things. A solder joint that's slightly different. An AC circuit nearby that changes things a little bit every 120th of a second. The baseline must be calibrated for every one of them. Once the signal has been converted to digital then it's OK. Errors smaller than the cutoff are ignored, and larger errors make one bit difference. For the calibration, you know what the outcome is supposed to be, so you set it to that.



    Could all this have somehow changed the outcomes so that some extremely unlikely results are falsely reported more often than they should be?



    There's no theoretical reason to expect it. And the engineers who assembled the LHC were very very careful. But how could we test it? The obvious way is to build at least 2 more LHCs and notice how consistent their results are. That would be very expensive. It will not be done.



    We can get some confidence by looking at results from other machinery. It's like -- the LHC was used to scan for a wide range of possible results that could be called the Higgs boson. They could do in years what a lesser machine might take centuries to do. But once we have a specific Higgs boson to look for, some of the others can look for that specifically and see whether they find it. If they do, then there's probably something there beyond equipment error.



    Something else they can do (which I think they are doing part of the time) is look for things that are supposed to not happen that nobody predicts will happen. When they find one for sure then everybody will get excited. People will say there's something wrong, and insist that they check for every possible error that could give them that result. Like with the faster-than-light thing.






    share|cite|improve this answer









    $endgroup$


















      4












      $begingroup$

      Any one experiment is repeated many times on the same equipment. They look for rare events, and it takes a lot of rare events to be sure that they aren't just coincidence.



      The question about how many LHCs it takes to be sure, is different.



      Each LHC component had to be carefully tested to make sure it was in spec. Remember the example of the experiment that seemed to get a result slightly faster than light. Because it was so important, they went to great expense to test everything, components around the world, until they found two components that were out of spec, that created the small error. If the error had been in the other direction would they have done that testing? No. They wouldn't even notice the error. It wouldn't be important. What made this one important was faster than light. Did they carefully record every out-of-spec component they found that would tend to slow the signal, that might cancel the positive errors they found? Maybe. That wasn't what they were looking for, though. That was a complication and not a solution to the problem.



      After the tested LHC components are installed they must be tested again in case they were changed while being handled.



      Then they must be calibrated. Every analog output could have a baseline that's a little bit off, because of random things. A solder joint that's slightly different. An AC circuit nearby that changes things a little bit every 120th of a second. The baseline must be calibrated for every one of them. Once the signal has been converted to digital then it's OK. Errors smaller than the cutoff are ignored, and larger errors make one bit difference. For the calibration, you know what the outcome is supposed to be, so you set it to that.



      Could all this have somehow changed the outcomes so that some extremely unlikely results are falsely reported more often than they should be?



      There's no theoretical reason to expect it. And the engineers who assembled the LHC were very very careful. But how could we test it? The obvious way is to build at least 2 more LHCs and notice how consistent their results are. That would be very expensive. It will not be done.



      We can get some confidence by looking at results from other machinery. It's like -- the LHC was used to scan for a wide range of possible results that could be called the Higgs boson. They could do in years what a lesser machine might take centuries to do. But once we have a specific Higgs boson to look for, some of the others can look for that specifically and see whether they find it. If they do, then there's probably something there beyond equipment error.



      Something else they can do (which I think they are doing part of the time) is look for things that are supposed to not happen that nobody predicts will happen. When they find one for sure then everybody will get excited. People will say there's something wrong, and insist that they check for every possible error that could give them that result. Like with the faster-than-light thing.






      share|cite|improve this answer









      $endgroup$
















        4












        4








        4





        $begingroup$

        Any one experiment is repeated many times on the same equipment. They look for rare events, and it takes a lot of rare events to be sure that they aren't just coincidence.



        The question about how many LHCs it takes to be sure, is different.



        Each LHC component had to be carefully tested to make sure it was in spec. Remember the example of the experiment that seemed to get a result slightly faster than light. Because it was so important, they went to great expense to test everything, components around the world, until they found two components that were out of spec, that created the small error. If the error had been in the other direction would they have done that testing? No. They wouldn't even notice the error. It wouldn't be important. What made this one important was faster than light. Did they carefully record every out-of-spec component they found that would tend to slow the signal, that might cancel the positive errors they found? Maybe. That wasn't what they were looking for, though. That was a complication and not a solution to the problem.



        After the tested LHC components are installed they must be tested again in case they were changed while being handled.



        Then they must be calibrated. Every analog output could have a baseline that's a little bit off, because of random things. A solder joint that's slightly different. An AC circuit nearby that changes things a little bit every 120th of a second. The baseline must be calibrated for every one of them. Once the signal has been converted to digital then it's OK. Errors smaller than the cutoff are ignored, and larger errors make one bit difference. For the calibration, you know what the outcome is supposed to be, so you set it to that.



        Could all this have somehow changed the outcomes so that some extremely unlikely results are falsely reported more often than they should be?



        There's no theoretical reason to expect it. And the engineers who assembled the LHC were very very careful. But how could we test it? The obvious way is to build at least 2 more LHCs and notice how consistent their results are. That would be very expensive. It will not be done.



        We can get some confidence by looking at results from other machinery. It's like -- the LHC was used to scan for a wide range of possible results that could be called the Higgs boson. They could do in years what a lesser machine might take centuries to do. But once we have a specific Higgs boson to look for, some of the others can look for that specifically and see whether they find it. If they do, then there's probably something there beyond equipment error.



        Something else they can do (which I think they are doing part of the time) is look for things that are supposed to not happen that nobody predicts will happen. When they find one for sure then everybody will get excited. People will say there's something wrong, and insist that they check for every possible error that could give them that result. Like with the faster-than-light thing.






        share|cite|improve this answer









        $endgroup$



        Any one experiment is repeated many times on the same equipment. They look for rare events, and it takes a lot of rare events to be sure that they aren't just coincidence.



        The question about how many LHCs it takes to be sure, is different.



        Each LHC component had to be carefully tested to make sure it was in spec. Remember the example of the experiment that seemed to get a result slightly faster than light. Because it was so important, they went to great expense to test everything, components around the world, until they found two components that were out of spec, that created the small error. If the error had been in the other direction would they have done that testing? No. They wouldn't even notice the error. It wouldn't be important. What made this one important was faster than light. Did they carefully record every out-of-spec component they found that would tend to slow the signal, that might cancel the positive errors they found? Maybe. That wasn't what they were looking for, though. That was a complication and not a solution to the problem.



        After the tested LHC components are installed they must be tested again in case they were changed while being handled.



        Then they must be calibrated. Every analog output could have a baseline that's a little bit off, because of random things. A solder joint that's slightly different. An AC circuit nearby that changes things a little bit every 120th of a second. The baseline must be calibrated for every one of them. Once the signal has been converted to digital then it's OK. Errors smaller than the cutoff are ignored, and larger errors make one bit difference. For the calibration, you know what the outcome is supposed to be, so you set it to that.



        Could all this have somehow changed the outcomes so that some extremely unlikely results are falsely reported more often than they should be?



        There's no theoretical reason to expect it. And the engineers who assembled the LHC were very very careful. But how could we test it? The obvious way is to build at least 2 more LHCs and notice how consistent their results are. That would be very expensive. It will not be done.



        We can get some confidence by looking at results from other machinery. It's like -- the LHC was used to scan for a wide range of possible results that could be called the Higgs boson. They could do in years what a lesser machine might take centuries to do. But once we have a specific Higgs boson to look for, some of the others can look for that specifically and see whether they find it. If they do, then there's probably something there beyond equipment error.



        Something else they can do (which I think they are doing part of the time) is look for things that are supposed to not happen that nobody predicts will happen. When they find one for sure then everybody will get excited. People will say there's something wrong, and insist that they check for every possible error that could give them that result. Like with the faster-than-light thing.







        share|cite|improve this answer












        share|cite|improve this answer



        share|cite|improve this answer










        answered 2 days ago









        J ThomasJ Thomas

        443210




        443210






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Physics Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            Use MathJax to format equations. MathJax reference.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fphysics.stackexchange.com%2fquestions%2f468891%2fhow-do-we-know-the-lhc-results-are-robust%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            數位音樂下載

            When can things happen in Etherscan, such as the picture below?

            格利澤436b