I was able to take the car for a pretty decent high speed 70+mph drive today with no issues.
I hope i'm not jumping the gun here. But fingers crossed it stays fixed and isn't some anomaly.
Thanks to Chris for putting me in touch with Jim who helped me diagnose the problem.
Jim and I double checked the valves and we took a ride. After hearing the problem he suggested I start by replacing the rockers.
I replaced them with a set I had off an old Midget engine and the car is running better.
The old rockers were missing a thrust washer (or at least a regular washer) on one end. That's the only thing I saw that was obviously wrong.
Hopefully it is fixed.
Thanks to all you folks for your help.
I'll update more later.