Check your Key-vault permission model under your Key Vault -> Settings -> Access Configuration on Azure portal.

Azure keyvault mainly allows key vault access using two permission models.

Azure role-based access control
Vault access policy

If you are using Azure role-based access control, make sure that you have given 'Key Vault Administrator' role to your service principal from Access Control (IAM) section of your key vault.

Refer: https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli

If you are using Vault access policy, make sure that you have added policy for your service principal with necessary permissions under Access Policies section of your key vault.

You could also use: 
Set-AzKeyVaultAccessPolicy -VaultName "<your-unique-keyvault-name>" -UserPrincipalName "[email protected]" -PermissionsToSecrets delete,get,list,set