I would agree with previous statements: avoid baking users into your base image with Packer. Likewise, you should avoid baking them with
Terraform as well. Terraform does not provide a mechanism to perform on-going changes after the server is provisioned. It only has those initial provisioning scripts that get run only one time. If you manage users with Terraform provisioning scripts, those users will never get updated again by Terraform. You might as well be baking these changes into Packer at that point.
I would honestly suggest you venture outside the Hashicorp ecosystem for this task. For on-going changes of machines, this is exactly what a configuration management system is designed for. Something like Anisble, Chef, Puppet, or Salt, real configuration management tools, are best suited for this job.
Packer lets you set a base image to be used by many machines. Terraform maintains the state of your architecture as code. However, what happens when you on-board or off-board new users? Or you need to change their individual SSH keys because they lost their computer? This should not be a code change. Nor would you want something like User secrets stored as code in your Terraform provisioning script.
At my company the stack is: Packer images, Terraform bootstrapping, and Chef management. They work well together. I'll admit, I love putting more things into Packer and Teraform as it makes spin up times quicker. But a good CM tool still has its place in the DevOps world.