You will probably have heard that we have found the Higgs boson - or something similar to it. We are not quite sure yet. You may also have heard that we found it in an experiment, and that this was a triumph for theory, which predicted it long ago. This seems to be a wonderful combination, theory and experiment. But, as always, nothing is just as simple as it seems.
Let us undertake the journey and accompany a theoretical idea from its inception until its experimental test, to see what is going on.
Having an idea of how physics beyond the standard model could look like is essentially simple. Though, of course, many ideas have already found by some of the people thinking about it since the early 1970ies. The interesting question after having an idea is, how to check, whether it is actually describing nature, or is just an interesting mathematical toy.
To do this, two things are necessary. The first is to check whether the idea is compatible with what we know so far about nature. The second is to use the idea to predict something which is different from the standard model. That is necessary, so that we can distinguish both, and decide how nature can be described. To do both we have to to somehow compare to an experiment.
Unfortunately, experiments cannot directly work with the mathematical stuff a theoretician writes down. Modern particle experiments work in the following way: You send something into a box and then detect what comes out of the box. In case of the suspected Higgs, we send in protons. The box is an empty space where these protons hit each other. Because the encounter is violent enough, everything comes apart, and out of the box come a lot of other (known) particles. These are then detected. Actually, we can pretty well by now not only say that there is a particle, but also what particle it is, and where it is headed with which speed. The set of detected particles is what we call an event. We then do many collisions and collect many events. The reason for this is that quantum physics forbids us to know precisely what is going on, but only what happens on the average. And to get an average, we have to average over many events.
At any rate, we end up with such information. That is what modern experiments do.
Now, the theoretician has to somehow convert his idea to something which can be compared to this experimental outcome.
In most cases, things roughly proceed as follows:
What we actually collide are not protons, but the quarks and gluons inside the proton. Thus, the theoretician first computes how quarks and gluons become converted into a new particle. Unfortunately, the experiment can only talk about the protons going into the box. So we have to first compute how we find quarks inside the proton. This is actually very complicated, and so far only partially solved. Nonetheless, we can do it sufficiently well for our purpose, though it is a challenging calculation.
The next problem is that the new particles lives usually only for a very short time. Too short to escape the box. It will decay into other particles before it can leave the box. In fact, it will often decay into particles, which in turn still do not live long enough to escape the box, but also decay first. So you have to calculate the whole chain of decays, until you reach particles, which are so stable that they will escape the box, and can be detected in the detector.
Once you have this, you have what we call a cross section. This is a number, which tells you how often two colliding protons will end up being a certain set of particles, which come from the decay chain of the new particle. Usually, you also know how often these particles go with which speed into which direction.
Unfortunately, we cannot yet compare to experiment, for two reasons.
The first is that the detector is not perfect. For example, the detector has to have a hole where the protons enter. Also, we cannot suspend the detector in thin air, and the holding devices produce blind spots. In addition, we are actually not able to measure all the speeds and directions perfectly. And it can happen that we mistake one particle for a particle of a different species. All of this is part of the so-called detector efficiency. An experimentalist can determine this with a great amount of work for a given detector. As a theoretician, we have to combine our prediction with this detector efficiency, to make a reliable prediction. Just think what would happen if our idea produces a signal which would escape preferentially along the way the protons came in. If we would not take the detector efficiency into account, we would just see nothing, and would decide our idea is wrong. But knowing the detector efficiency, we can figure out what is going on.
The second problem is what we call background. This background has two origins.
One is that the remainder quarks and gluons of the protons usually do not go away nicely, but will produce many other particles in other collisions. At LHC we even have that usually more than two protons collide. This produces a lot of debris in the detector. To find the new particle then means to separate all the debris from the particles into which the new particle has decayed.
The second problem is that other processes may mimic the searched for particle to some extent. For example, by having similar decay products. Then we have to distinguish both cases.
Because of the detector efficiency, we are not able to resole both types of background perfectly. And neither can we resolve the signal perfectly. We just get a big pileup, and have to find the signal in it. To do this, theoreticians have to calculate all this background. By comparing than what one gets from background alone and from background plus the desired signal, we have reached our goal: We know, how our searched for new particle appears in the detector. And how the experiment will look like, if we were incorrect, and just the known bunch of things is there.
All of this is quite laborious, and a lot of groundwork. And all too often it comes out that for a given detector efficiency we will not be able to get the signal out of the background. Or that the number of times we have to try is so large that we cannot afford it - we would just have to get too many events to get a reasonable reliable average.
But well, this is life. Your options are then either to wait for a better experiment (which will usually take a couple of decades to build), or to go back to the drawing board. And find a new signal, which the present experiment can find. And then it may still happen that they find nothing, and this means you idea was incorrect from the very beginning. And then you can go back to step one. In case of the Higgs, it appears likely that it turned out be correct. But there the process was so complicated that it took 48 years to have good enough experiments and reliably enough theory. Physics beyond the standard model may require even more, if we are unlucky. If we are lucky, we may have something in a few months.