You are correct - what you'll want is a Switch most likely.
When you say you have 4 internet ports on your modem - is this the device that you plug into the line? Are you able to access the internet on multiple devices already?
Many routers have an inbuilt modem but sometimes it's a separate device. I suspect if you have 4 ports on it then you do indeed have a router in which case an Ethernet switch will give you more ports easily. Just plug one cable between the switch and the router, and the remaining vacant ports on the switch will be available for you to use.