Text this: Gradient-Based Algorithms With Intermediate Observations in Static and Differential Games