The goal here is to make this a repeatable process for any and all alerts. First let’s describe the general validation process you should go through when picking an alert:
- Is there an absolute limit to alert on, regardless of context? (usually the answer is no, or if it is yes, you will generally want to know at some point prior to this happening)
- Determine what is normal (baseline)
- Determine what is worrying (warning)
- Determine what is a definite problem (critical)
- Determine the likelihood of false positives
Read more here