the definition of the replicated pods are always a copy-paste of a pod template file
This behavior is implemented by Kubernetes itself: it takes the template
part of the deployment spec and creates replicas
identical pods.
The corollary to this is that you don't need to manually write out the pod spec as a separate object. Including it in the template
part of a deployment (job, statefulset, daemonset) is enough.
Is it possible to rather use a reference to the yaml file (or an url) defining a pod
No, individual Kubernetes objects must be self-contained (except where there are facilities to do things like inject Secret values into environment variables). The Kubernetes server doesn't know about your local file system.
Using one of the templating systems out there (Helm is more-or-less standard but not the only choice) could help reduce this duplication, if you really need near-identical deployments and also bare pods. You could write something like
---
apiVersion: v1
kind: Pod
{{ template "pod-spec" . }}
---
apiVersion: apps/v1
kind: Deployment
spec:
template:
{{ include "pod-spec" . | trim | indent 4 }}