we have two independent kubernetes clusters that work like Main and it's high available one. and we have some stateful applications that store data within cluster's persistent volumes. my question is .. can I share a PV between those two clusters, so I can have a final result of two pods running on the two clusters reading/writing on the same volume as if they're running on one same cluster. I'm focusing on sharing data between the pods regardless how the pods would handle the data writing.
Can you share a PV, between clusters: in theory yes. You can mirror one PV object into another cluster, given you have similar storage classes and accesses to your storage backend.
Sharing PV between Pods, within one or more clusters: depends. First, on your storage backend. There may be some locking mechanism that would prevent you from attaching the same volume to more than one client at a time.
In addition of which: if your storage backend provides with block devices, your volumes would involve some file system, that may not support being mounted on two systems at once.
To work around those two points above, we could consider something like NFS, CephFS, EFS (aws), a samba share, ... Next, it depends on your application: what are the risks of those two Pods of yours writing the same file at once, and corrupting your data?
Short answer: in theory yes, you could. Although if possible: try to question this requirement. Sounds unusual, may be something wrong with design/architecture.