I use Ubuntu. Currently Feisty, but I'll probably upgrade to Gutsy after it hits the 1-month old mark!
My server is not public - it is local, but it mirrors my public server and works quite well. It's a LAMP server.
I have phpMyAdmin set up to administrate my databases.
Postfix is the mail server I use, it's very intelligent
I also have Ruby installed because I'm learning Ruby.
I have Wine installed and via Wine I have IE5, IE5.5, and IE6 installed. This allows me to do cross-browser checks on my pages.
I use FileZilla's Linux build for my ftp client since I upload to a remote host after I've developed on my local server.
I use Bluefish for all HTML and PHP editing. It is a good basic CSS editor as well, however, I have my Linux box networked to my XP box via Samba to use Stylemaster CSS editor. It's really wonderful and I prefer to use it to just Bluefish. But Bluefish is great for all other web development, imo. If you write your code from scratch.
Samba is also useful because it allows me to use IE7 on my Windows box to do cross-browser checking.
As far as forums, I've used bbPress for a client site and I like it. It's clean and light. Doesn't quite as involved as phpBB so you really need to pick what you/your client needs.
I set up tons of Wordpress blogs and have my own Wordpress blog running on localhost (not ready for the world to see it heh).
I've also set up Gallery as a photo gallery script for a client, it's superb.
I have used Infinite Autoresponder script on one of my own sites and like it
And I'm just now getting my feet wet with Zencart for a client that needs a shopping cart and so far I'm liking it.
I love web development. CSS especially gives me sweet warm fuzzies inside
Good luck in your endeavor! I loved setting up my Linux based server and tweaking it around (my remote host is also a Linux server!)
Oh, editing to add I use the GIMP for all image manipulation. There aren't as many tutorials as there are for Photoshop thus it takes more work to get into but it's a really powerful program. Akkana Peck's book is wonderful if you want to dig into the GIMP as you develop!