MP4 Smooth Review Problem

I am using Pr CS5.0

I shoot entire soccer games (hour long) and save them as MP4 files.

At the end of the season I make a highlight video by importing all the MP4 files and editing them down to one 30 minute video.

I fast forward through each game using the short cut keys looking for good plays.

When I try to reverse (j key) there is always a 3 second hesitation before the video actually starts playing backwards.

Also, most of the MP4 files play well during Pr review but some of them have frequrnt periods where the video is jumpy.

The resulting video or DVD play in not jumpy.

Both of these problems happen for both HD and standard def MP4 files.