Simply having your computers connected to a router for Internet access does not mean that they are able to access each others files, or even know the others exist. There are further steps that need to be taken.
The first thing you need to do is run the Windows Network Setup Wizard from the Control Panel. It will help you in setting up unique names for every computer on your network, and adding them to a workgroup. You will choose the setting for "This computer connects to the Internet through a Gateway..." and you need to choose to enable file and printer sharing.
The next step is to create identical user accounts on every machine. For example, if you have four computers and your name is Joe, you should create a user account on each computer called Joe, and use the same password. Leaving the password blank is not a good idea. You can name the user account whatever you'd like, but it needs to be identical on every machine, as does the password. Names like "Guest" or "Administrator" cannot be used, as they are default account names in Windows.
After you have done these steps, you should be able to access all the computers in your workgroup from My Network Places.