One of the first D&D houserules I encountered was the “crit to kill” rule: if you roll a natural 20 on an attack, roll another d20. On a second 20, the guy dies.
In Jon Peterson’s Playing at the World, I found a reference to a 1975 ancestor of that houserule, published in 1975 by Gary Switzer, the guy who wrote the first version of the Thief class. Peterson says, “On each melee swing, the attacker rolls an additional d20, which if it scores a ‘0’ (bearing in mind that early d20s had two 0’s), results in a critical. It is the main attack die which determines whether this is a critical hit or a trip—if the attack roll succeeds, then a hit has occurred, otherwise it is a trip.”
That extra die roll never made it into D&D canon, but 3e introduced the idea of “confirming crits:” rolling to see if your natural 20 was really a crit or not. (I think it introduced it. As I’ve mentioned before, my weakness is 2nd edition rules.)
I don’t like any of these rules very much in actual play, but not for game balance reasons. Sure, an insta-kill on 1 in 400 attacks adds wackiness and mitigates against PC survivability, and confirming crits only helps in bizarre corner cases where goblins crit on 100% of hits against dragons. The reason I don’t like them, though, is because they add anticlimax rolls.
In the “crit to kill” variant: You crit! Roll another d20. On a 1-19: Oh well, at least I got a crit. With the confirming a crit rule: I rolled a 20, but failed on the confirmation roll! Oh well, my crit didn’t happen.
4e’s solution was to have a crit always do max damage, and then throw some extra damage dice into the mix. That was not a bad solution: the extra dice always felt like bonus damage, even if you rolled poorly.
I COULD imagine bringing back the “crit to kill” rule in a modified form.
Recently, I’ve experimented with rolling d20s (and even d100s!) for special-effect damage. It’s fun! In my proposed variant, when you roll a crit, you don’t double or max your damage; you throw in a d20 along with all your other damage dice. If that die rolls a 20, you insta-kill. Otherwise, it just adds its damage to your hit (an average of 10 extra damage).
If the 1-in-400 chance of an insta-kill is too silly for you, you could instead make it an exploding d20 roll: on a crit-die roll of 20, you add 20 damage and roll again. Against all but high-level opponents, it will come to the same thing. However, this tweak gives 20th-level fighters some protection against 400 goblin archers.