when you set up your Business Hours, you can set the option "Generate alerts (performance) during business hours only" (see attachement).
As Far as I know, this is the only option you have and there is no further possibility to restrict alerts for specific time-slots. Nonetheless (when you activate these checkbox), the alert should appear in the Alert Log of the CAS.
Apart from everything what has been said already, for your use case I would also consider creating filters on dimension "Hour of the day" (and "Day of the week" if you like). Those dimensions are available in Dynatrace NAM alerts mechanism starting from release 2018. See example configuration: