Text this: Physics-informed Priors Improve Gravitational-wave Constraints on Neutron-star Matter