It's likely that you'll eventually find that it will be impossible to keep the plugins truly version agnostic.
Somewhere along the line one version of the regular code will make an assumption about the plugins that happens to be correct in one version, but changes in a later version. Even if you fix all of your branches to work with newer versions of the plugins, historical commits in those branches won't work with the newest plugins. This can make it difficult to some kinds of debugging (eg: anything with
Instead of having the plugins only in the latest branch, it might be better to have them in all of the branches, and have a policy that changes to the plugins happen only in the latest version branch, and are then "backported" to the older versions' branches by cherry-picking the plugin update commits.
However, the best is probably to move the plugins into a separate repo and either use a package manager or git submodules. Then each version branch can be explicit about which version of the plugins it works with.