A wallet is not ‘stored on the blockchain’ the way a bank account is — funds sit in outputs locked to conditions. Your private key is the secret that satisfies those conditions for spends; the public key condenses into an address peers use to pay you. Lose the private key and the coins remain on-chain forever but unspendable; leak it and anyone can move them first.
What each key does