Which provides better performance - a clock source with sinewave output, or one with differential square wave outputs?

An important factor in reducing jitter is the slew rate of the input signal. PECL outputs tend toward a square waveform, which, along with their typical 800 mV amplitude, have good slew rates. In addition, PECL is usually used differentially, which improves the slew rate further. This makes PECL a good choice for a clock source.

However, some clock sources are available only in sinewave output versions. Sinewave outputs can certainly be used, either differential (when available) or single-ended. With sinewave signals, the slew rate is dependent on the frequency and the amplitude of the signal. For a given frequency, a higher amplitude signal has a higher slew rate. The amplitude of a sinewave clock source is therefore of importance. Often, the output of a sinewave source is specified in dBm, which is a power output. This can be converted to a voltage swing if the impedance of the operating environment is known.

So, it is OK to use either a differential square wave, or a sinewave, clock source. Slew rate is the key to improved jitter performance, all other things being equal.