The number one issue with submersion systems is the question of what kind of "case" to use. A standard PC case is absolutely out of the question - no amount of caulking could seal one of those up. To find a case for a submersion system you have to get a bit more creative. Fish tanks are a very popular solution, and this fact has led to many aquarium inspired submersion PC mods. (It's a shame fish can't live in mineral oil).
With a submersion system, you're also going to need a pump and a cooling device - usually a radiator, but sometimes more interesting methods are used. The pump is required in order to both circulate the oil as well as run it to the radiator. Occasionally, some sort of filter lies between the two, but most won't bother with a filter when the oil only really needs to be cleaned once a year or so. The radiators used in submersion systems are often very large and highly customized. Sometimes a rather expensive passive radiator can be used, like this Zalman.
There is an interesting alternative to the pump and radiator method that involves using an Air Conditioning unit coil submerged along with the rest of the PC, that is reported to produce 0c temperatures (yes, freezing). This isn't well documented, is quite bulky, ugly, and expensive, but it sure does get the job done!