opponent had an ancient tomb in play. I casted primal command choosing the modes “Put target non-creature permanent on top of it's owner's library.” and “Shuffle target player's graveyard into his or her library.”

First mode targeted opponent's Ancient Tomb. Second mode targeted opponent, even though opponent had no cards in graveyard. No shuffle animation played, and opponent drew and replayed Ancient Tomb.

This is not how this card works. If the shuffle mode is chosen and resolves, the opponent HAS to shuffle their deck, even if there are no cards in it. The mode combination chosen is a classic way to use Primal Command to effectively remove a key target.