Uncorrelated shape: shapesys vs staterror ?

[zjkjsd]

Dear experts,

There are 2 modifiers in pyhf for uncorrelated shape uncertainty: shapesys and staterror, and I am confused about which one to use (though we know one is constrained by Poisson and the other by Gauss).

For example, I have 10 samples in my SR channel, let's call one of them my_sample. If my_sample is from MC and I know it has 4 different sources of uncorrelated shape uncertainty. My questions are:

1. How do I know which modifier to use? (Any suggestions/examples are appreciated, it's also helpful to see how other people choose, don't have to answer to my case specifically)
2. If I choose shapesys for 2 sources and staterror for the other 2. Do I add the modifiers one by one? i.e. shapesys_1, shapesys_2, staterror_1, staterror_2. Or do I need to combine them? i.e. shapesys = source_1 + source_2, staterror = source_3 + source_4.
3. How should we calculate the input if the modifiers can be combined?

[alexander-held]

The staterror modifier is special in the way it acts on groups of samples: all assigned samples vary in the same way, while for shapesys the variation is defined per sample and when the assigned parameter for a bin moves around, the associated samples will all change but you can still define per sample what the change is.

1. The typical use case for staterror are MC statistical uncertainties in the Barlow-Beeston "lite" method. I cannot think of very typical shapesys use cases at the moment, this is a fairly rare modifier type in my experience. Here is an example for using shapesys to split out contributions from the total MC statistical uncertainties per sample: About signal statistical uncertainty cabinetry#450.
2. I am not sure what you mean here. staterror combines behind the scenes so everything it acts on will be moving up/down in the exact same way. shapesys has no such mechanism. Whether you want to combine multiple sources of uncertainties together is a question for how you want to set up your model.
3. Also not clear to me what you mean here.

Note that it is currently possible to assign multiple staterror modifiers to the same bin but I would view this as a bug and not rely on it, see #1830. If you need more bin-by-bin uncorrelated sources of uncertainty implemented, shapesys is the way to go.

[zjkjsd]

Thank you very much, Alex! This is helpful as always! Let me clarify my Q2 and Q3 (though you have answered it a bit in your Note):

1. For a sample, e.g. signal sample, if I have 4 different sources for uncorrelated shape uncertainty, should I add them one by one in the modifier list like below?

"channels": [
        {
            "name": "ch1",
            "samples": [
                {
                    "data": [1000.0],
                    "modifiers": [
                        {"data": null, "name": "mu_sig", "type": "normfactor"},
                        {"data": [10.0], "name": "staterror_source_1", "type": "staterror"},
                        {"data": [20.0], "name": "staterror_source_2", "type": "staterror"},
                        {"data": [30.0], "name": "shapesys_source_3", "type": "shapesys"},
                        {"data": [40.0], "name": "shapesys_source_4", "type": "shapesys"}
                    ],
                    "name": "signal"
                }
            ]
        }
    ]

2. Or is it only allowed to use one type of modifier once for a sample? (it sounds like so in your note). If this is the case, is the config below correct?
3. How do I calculate the input for the combined shapesys and staterror, I simply added the source_1 and source_2 as 10+20, but not sure if it is correct.

"channels": [
        {
            "name": "ch1",
            "samples": [
                {
                    "data": [1000.0],
                    "modifiers": [
                        {"data": null, "name": "mu_sig", "type": "normfactor"},
                        {"data": [10.0+20], "name": "staterror_source_1+2", "type": "staterror"},
                        # {"data": [20.0], "name": "staterror_source_2", "type": "staterror"},
                        {"data": [30.0+40], "name": "shapesys_source_3+4", "type": "shapesys"},
                        # {"data": [40.0], "name": "shapesys_source_4", "type": "shapesys"}
                    ],
                    "name": "signal"
                }
            ]
        }
    ]

4. Or can we combine all the uncorrelated shape uncertainty into a single modifier, like below? Does it make sense?
5. What modifier to choose to be the final modifier? (if it makes sense)
6. How do I calculate the input for this case? (if it makes sense)

"channels": [
        {
            "name": "ch1",
            "samples": [
                {
                    "data": [1000.0],
                    "modifiers": [
                        {"data": null, "name": "mu_sig", "type": "normfactor"},
                        {"data": [10.0+20+30+40], "name": "shapesys_source_1+2+3+4", "type": "shapesys"},
                        # {"data": [20.0], "name": "staterror_source_2", "type": "staterror"},
                        # {"data": [30.0+40], "name": "shapesys_source_3+4", "type": "shapesys"},
                        # {"data": [40.0], "name": "shapesys_source_4", "type": "shapesys"}
                    ],
                    "name": "signal"
                }
            ]
        }
    ]

[alexander-held]

1. This is a question I cannot answer in general, it is really up to you. If you have different sources of uncertainty that are in no way correlated you might as well combine them but the result should not be affected. Note that two staterror on the same sample are not a good idea (see previous reply). The shapesys numbers should sum in quadrature, you can confirm that by giving it a try (if you do it correctly, you will see the fit results are unchanged).
2. No, you can use many different modifier types on the same sample. What I talked about is that you should not use two staterror on the same sample.
3. See 1., they should sum in quadrature so e.g. $\sqrt{10^2 + 20^2}$. Note again that you should use shapesys here.
4. You can, but as I said before: this is your choice.
5. shapesys
6. Sum in quadrature, but try it out: if the result is unchanged you're doing it right.

[zjkjsd]

To summarize what I learned in this discussion:

1. Use staterror for the MC statistical uncertainty, all samples vary in the same way in a bin, i.e. one parameter per bin, which makes sense to represent statistical effects.
2. If there are more sources for uncorrelated shape uncertainty, consider using one or more shapesys modifiers. For example, if there are additional 2 sources:

"channels": [
        {
            "name": "ch1",
            "samples": [
                {
                    "data": [1000.0],
                    "modifiers": [
                        {"data": null, "name": "mu_sig", "type": "normfactor"},
                        {"data": [31.6], "name": "statistical_uncertainty_ch1", "type": "staterror"},
                        {"data": [10.0], "name": "uncorr_source_2", "type": "shapesys"},
                        {"data": [20.0], "name": "uncorr_source_3", "type": "shapesys"}
                    ],
                    "name": "signal"
                }
            ]
        }
    ]

3. If the 2 additional uncorrelated shape uncertainties are completely independent, they can possibly be combined with their input to be summed in quadrature (will test it myself) like below:

"channels": [
        {
            "name": "ch1",
            "samples": [
                {
                    "data": [1000.0],
                    "modifiers": [
                        {"data": null, "name": "mu_sig", "type": "normfactor"},
                        {"data": [31.6], "name": "statistical_uncertainty_ch1", "type": "staterror"},
                        {"data": [sqrt(10^2+20^2)], "name": "shapesys_source_2+3", "type": "shapesys"},
                        # {"data": [20.0], "name": "shapesys_source_3", "type": "shapesys"}
                    ],
                    "name": "signal"
                }
            ]
        }
    ]