Most guinea pigs love spending quality time with their owners, so handling them is enjoyable for all concerned. Once your guinea pig gets to know you, being hand-fed and petted become important parts of their lives. This means it's actually up to you how much you hold them - you can do so as often as you like. But even if you decide to have hands-on sessions infrequently, you'll still need to groom them on a regular basis, and carry out the usual health checks.
Guinea pigs love human contact, once they get to know you
Don't handle newborn guinea pigs at all. The same applies to new guinea pig moms, or any GP that has just returned from the vet, or is ill in any way. Leave the pups and the moms for at least a week, and ask your vet when you can start handling a guinea pig who has been unwell or had an operation.