Given how often ship-to-ship combat appears in adventure fiction, it's surprising how short a shrift it gets in D&D. In most editions, it takes secondary importance to more esoteric situations like aerial combat and underwater combat. When your ship is menaced by pirates and it's time to break out the ballistas, it's often hard to find the relevant rules.
Here's how naval combat has fared in the various editions.
OD&D: For its page count, OD&D is one of the most comprehensive RPGs of all time. After 3 1/2 pages on aerial combat, we get a luxurious 8 pages on naval combat. Boarding, ramming, shearing oars, and maneuvering are covered pretty clearly, and we have very complete stats on each ship type's crew and its speed in various wind conditions. Missile fire is the least developed aspect: "All missile fire, including the various forms of catapult fire, are as in CHAINMAIL. Catapult hits will do points of damage to ships, and when sufficient points have been scored the ship sinks. Large ships have from 18-24 points of possible damage before sinking, small ships have from 9-15, and a boat but 3 points."
Let's look in CHAINMAIL. The relevant rules are those for catapults besieging castles. A light catapult does exactly 2 points of damage so it will take 9 to 12 hits to sink a large ship. How do you hit?
1) Set up the battle on a table.
1) Estimate how many inches away your target is on the table, without measuring, and declare your attack range.
2) Cut out a circular plastic disc, 2 inches wide, and put it that many inches away from your catapult. If it overlaps the ship mini, you may inflict your 2 points of damage.
1st Edition: Naval rules get 2 1/2 pages in the DMG, right between aerial combat and underwater combat. First we get an overview of typical D&D ships, which include the drakkar, cog, and caravel, but "the ultimate warship for the purpose of AD&D is the nao" (also called a carrack). We're trending towards the 14th century here. Although D&D is gunpowder-free, the nao is armed: The crew "consists of 2 or 3 men to work each ballista, 3 or 4 men to handle the catapult and the rest to man the sails." Each ship type gets its own hit point range: 1-4 for a rowboat and 7-42 for a warship like the nao.
Now on to the combat rules! There's plenty of detail on speed and wind direction, and a full column on handling fires. Boarding rules are the usual but totally adequate "use regular D&D combat rules". The ramming rules are a little undercooked: apart from the fact that "you must strike the other ship at a 60-90 degree angle," there's no defense and no reference made to the HP of the ship. I guess it always works: "Depending on the size of the ship and the location and size of the hole, it may take from 1-12 turns before she sinks below the surface of the water."
Despite all the ballistae and catapults mentioned in the ship descriptions, we get one sentence on how to use them, halfway through the "Sinking a Ship" section: "It will take several direct hits with a boulder before enough damage is done to cause a merchant ship or a warship to sink (see Hull Values and Siege Engines.)" Once again we have to consult the castle siege section of the rules to adjudicate naval combat.
Hit determination is fairly simple: ballistae consider everything AC 10, while catapults consider everything AC 0. This is modified for range, wind, size of the target (large ship: +4), etc. Damage is a little tougher. There's a matrix which compares every material type (wood, hard stone, etc) with every attack type, from Bigby's Clenched Fist to Trebuchet. A ship is presumably a wooden structure, so a small catapult does exactly 4 points of damage per hit. It will take about 7 hits to sink a warship. Despite the specific rules for a ballista to-hit roll, there is no damage entry for a ballista.
Basic D&D: The Expert rulebook has a tidy page on Combat at Sea. Ships have speed, AC (from 7 to 9), and Hull Points (similar to HP; a war galley has around 135). "Unless noted otherwise, most giant sea creatures and certain magic attacks will inflict 1 point of hull damage for every 5 points of normal damage."
The rules cover catapults but not ballistas (catapults attack as a fighter of a level equal to the number of catapult operators and do 3-18 hull damage), fire (1d6 damage a turn), ramming (attacks as a first level fighter, does as much as 110 hull damage), and boarding (use your existing melee rules). Clearest rules so far.
2nd Edition: 2E has no naval combat rules in the core rulebooks. The DMG has 4 pages on aerial combat and 2 pages on underwater combat, and a section on siege engines which is remarkably inapplicable to ship-to-ship combat: instead of doing HP of damage, each hit removes cubic feet of material from a wall.
The first ship-to-ship rules are in the 1992 Forgotten Realms supplement Pirates of the Fallen Stars. My long-held belief is that if your edition's only rules for something are in a Forgotten Realms book, your edition has no rules for that thing.
The best 2e rules for naval combat are probably in Spelljammer. Ignore the fact that it's in space: Spelljammer ships have speed, maneuverability class, facing, armor rating (AC) and hull points (HP). Weapons have a THAC0, variable damage, and range. For the first time, everything I need to resolve a ballista attack, all in one place!
Spelljammer also has charts for damage to specific parts of the ship, a page and a half on ramming, a column on shearing, and 2 pages on boarding. Instead of saying "use D&D melee rules for boarding," it presents a mass combat subsystem. And, despite the fact that it's in space, combat is in 2D!
2E went on to print large-scale naval combat rules in a Birthright setting sourcebook, and, at the end of its run, "Of Ships and the Sea" in 1997. I haven't read the latter, but I understand that the rules are complete but cumbersome. If anyone's got it, can they chime in?
3rd Edition: 3E has no naval combat in the core books. The PHB includes speed for various ships. The DMG includes rules for aerial combat and underwater combat.
Vehicle combat (including ships) appears in the 3.0 Arms and Equipment Guide. Instead of ships having hit points or hull points as in previous editions, each ship hull is divided onto many 10x10x10 sections, and the masts are divided into rigging sections. Each has its own hardness and HP. Destroying sections reduces ship effectiveness, and also damages adjoining sections next turn, in a cascade of death that might take several rounds to adjudicate.
For missile fire we're again referred to the siege section of the DMG, where we learn that 3e treats a ballista, hilariously, as a Huge crossbow using the weapon size rules, so medium creatures have a -4 to aim it and Small creatures have a -6. It takes about 3 or 4 skill checks to load and fire each siege engine.
The major 3.5 sourcebook for nautical adventure is Stormwrack. Stormwrack comes with an apology for its rules: "The best way to keep your D&D game running smoothly during a ship-to-ship encounter is to make any naval battle in which the PCs participate into a boarding action as soon as possible." In third edition, that might be true.
Stormwrack replaces battlemat combat with a narrative system where only the distance between two ships is important (so it doesn't work for 3 or more ships.) Instead of sailing around each other, captains make opposed Profession (sailor) checks for "the advantage." You can't do anything without The Advantage except try to steal The Advantage, which is also done with the captains' opposed Profession (sailor) checks. Ramming, grappling, and shearing are all opposed Profession (sailor) checks that you can make if you have advantage. Your Profession (sailor) modifier basically determines the outcome of combat. It's like if D&D combat used your initiative roll for initiative, to-hit, and armor class, and if you lost initiative you'd never get to take a turn at all.
4th Edition: No naval combat rules in the core rulebooks (though, as usual, the DMG has rules for underwater and aerial combat).
Vehicles (including ships) are detailed in Adventurer's Vault. Vehicles have HP, AC and defenses, so they interact normally with most game rules; HP is scaled to characters, so a "greatship," for instance, has 400 HP. Vehicles also have facing, which isn't otherwise a feature of 4e, and, regardless of type, have a turn radius of 90 degrees per turn. Adventurer's Vault includes rules for crashing and ramming, but I can't find standard rules for ballista or catapult fire anywhere, even in the siege section of the DMG, since there are no siege rules in 4e.
Conclusions: Naval combat is usually an afterthought in D&D. It hasn't cracked the core rulebooks since 1979. When it appears, it's either incomplete or abstruse. The most elegant and usable rules are, by far, the 1-page rules in the Basic set, with Spelljammer the surprise runner-up. The worst set of rules is 3rd edition, with its honeycomb of interrelated hull sections and its overreliance on one skill.
Next week I'll write up my own one-page, any-edition naval combat ruleset.