If you're going to go the infrastructure-as-code way, you're going to want to store that code in a repository. Having your infrastructure definition stored in a VCS (GitLab) has pretty much the same security implications as storing code in GitLab. Just make sure you don't store sensitive data or credentials in plain text, or at all if you can help it. In Ansible's case, that usually means using Ansible Vault to encrypt sensitive files. But that's by no means the only way.
As for deploying, you can use ansible-pull, which runs on the machine you want to set up:
ansible-pull -U <repository> [options] [<playbook.yml>]
It will get the Ansible config from the specified repository to local host, then executes the playbook you tell it to.
That should meat your requirements, but it also means that the machine executing
ansible-pull will have access to everything that is in that repository. Whether that is secure enough for your environment or not is up to you. You can split your Ansible config in several repositories, use different branches, revoke the machine's access after the initial pull, etc.
In general, encrypt sensitive data in the repository (encryption on disk), make sure you pull over SSH or HTTPS (encryption in flight), and be mindful of who has access to the repository, even indirectly (access/user management), and you've gone a long way towards having a decently secure environment.
Whether using CI to deploy Ansible is secure enough or not, that's debatable. One could argue that someone who gains (write) access to the Travis/GitLab account where you keep your Ansible playbooks already got access to your infrastructure since they can alter those playbooks, it's just a matter of time. On the other hand, code stored in repository is half the point of infrastructure-as-code. YMMV.