I haven’t noticed any percievable speed differences between the two modes, although there are certain cases where you might want to avoid this mode.
There are actually significant, effectively fatal performance problems (such as 30-second loading delays, or longer) on mobile devices when using optimizeLegibility for long pages. Apply it only if you know what the maximum text length will be. (Also, avoid using it for Android clients, at least on the older versions that everyone still uses: its font renderer often has very strange bugs when this mode is enabled.)
I did some testing with Instapaper to determine approximate optimizeLegibility performance limits. A 5,000-word article in Instapaper for iOS, for instance, will only use optimizeLegibility on devices with an A5-class or greater CPU. To avoid problems on older iOS devices, I wouldn’t recommend using optimizeLegibility blindly and unconditionally on any pages longer than about 1,000 words. And I wouldn’t recommend enabling it on Android at all.