The following command works, cobbled together from friendly advice and additional googling based on that input:
podman build --ulimit=nofile=131072:1048576 [other arguments here]
The latest releases of podman apparently accept --ulimit=host, which is convenient if you want the same policy, but 3.4.4 does not. For 3.4.4, you'll need to know the specific limit name. These are not yet listed in podman documentation, but for the file descriptor limit the proper name is nofiles. This Red Hat article may be helpful with regard to other limit names.