It also depends on your security model

You should keep in mind that at least in theory, Spectre can be exploited remotely: Send crafted network packets to train the speculation engine and afterwards extract information based on timing of future packets.

I do not expect this to be a problem right now, as it will be even harder than a local exploit and the bandwidth of the hidden channel will be much lower than for a local attacker. The SNR will be much worse, due to increased noise. But it is not as easy as saying "If $networkprocess is broken, then we have lost anyway". Spectre works even without breaking any process.