The decline of civilization is driven by 2 types of people:Destroyers an Enablers.
Destroyers are quite aggressive towards civilization.They can be foreign (barbarian invasion) as well as domestic (thugs).Healthy civilizations usually eliminate thugs,which leads to genes correlating with crime being removed (genetic pacification).If thugs are not controlled,then they reproduce more offspring with thug mutations and result in deterioration of social relations.
Enablers are,on other hand,"civilized" kind of people who are incapable of committing lesser violence to prevent major violence and have their in-group preference turned off.They pity the Destroyers and actively help and protect them from people who want to eliminate Destroyers.
Enablers can be easily manipulated by infiltrators into destroying their own society.