I made a small animation showing the effect of small focus shifts, see below. The subject is Lincoln's jaw on a 1954-S uncirculated Cent. It's made from 4 images taken at slightly different focal planes, sort of like a stack, but within the depth of field. Mag is ~0.7x. The crops were taken from center of frame, and the lens was a 105PN at f2.8, so no expectation of CA issues. There is also no real change in perspective between shots, or angle of lighting, since the working distance is ~10in, and I'm only moving focus ~20um per step.

I'm sure there is a good explanation for this phenomenon, I just don't know what it is.
Edited to add:
Here's a 200% crop of the area that seems to distort the most:
