The type of server you decide on will determine the software you want installed. If you want a file server, you will want Samba and ssh installed. A print server will need Samba and CUPS. Apache HTTP server will be needed if you want a web server.
Unless you are creating an application server, you will not need office suites, games, graphic software, etc. It is a good idea to remove these applications. Removing the unnecessary applications deters other users from using the computer for something other than what it's designated for. It also frees up resources and disk space. When you are finished configuring the software, you should even turn off the X-server. You can always turn it back on if you need to.