Text this: Nonlinear optical excitation response and regulation of instabilities in a Y-type atomic system under complex interactions