Any time I see this issue on our tablets, it has always been the pen tip!
Some have even looked fine till you press the tip and then see the plastic is split. Try changing the Pen tip and bet that clears up your issues!
Otherwise 21 Surface Pro 4s and all write perfect lines till the tips go bad.
Ok , not good.
Have watched a few take the 4' to 5' fall to the solid floor. In the past that killed the pens for the other tablets, now it is just the tips.
By chance, did your pen ever take a straight drop to the floor with the solid tip in place?