I tried the bash completion stuff, and found a single thing I liked, ssh completion. Most of the other stuff just annoyed me… a lot (and was mostly useless, at least for me). I first looked at the code in the bash completion for ssh completion, but it was imho too large to stuff into my .bashrc, so I wrote a small perl one-liner that did the trick.

complete -A hostname -C "/usr/bin/perl -e 'my \$match = \$ARGV[1] ? \$ARGV[1] : \".*\"; open(my \$INPUT,\"<\",\"$HOME/.ssh/config\") or exit(0); foreach(<\$INPUT>) {next unless s/^\s*Host(name)?\s+//;chomp;foreach(split(/\s+/)) { print \"\$_\\n\" if(/^\$match/ and not /^\d/ and not /\*/);}}'" ssh 

Stuff that into ~/.bashrc and off you go. It completes based upon hosts in ~/.ssh/config