Text this: Probing the effects of broken symmetries in machine learning