Text this: Frozen Weights as Prior for Parameter-Efficient Fine-Tuning