Your router should have a DHCP server built in, unless it's a hub you're talking about.... anyways you don't need a DHCP server, what the DHCP does is assign a dynamic IP to each PC which you don't need, just enter a static IP on your TCP/IP settings on network properties and enter your ISP's DNS servers and you're set.
If you got a hub then you either need to pay your ISP for another IP for each PC or setup a DHCP server, for that you would need to connect the modem to the PC that will be the DHCP server, configure the DHCP server on windows and connect the other PCs thru the hub... but still it would be better for you to go to Best Buy and spend $20-$30 on a router.
If you need more assistance please post your router's make and model and explain how you're networking your PCs, and exactly what you want to do.
<edit> ah and also let us know what kind of connection to the internet you're using..... if it's a dialup modem then it's a little different...
<edit> and if not dial up let us know whether you use a USB modem or a NIC card.
Edited by fernan82