There are several related system engineering trade-offs and choices involved in answering your question. It may be helpful going forward to tackle this question by first prioritizing your objectives and then making the selection and trade-off decisions about VLAN structure, cluster(pod) sizes, etc. to decide which options give you the most bang for the buck in vertical scaling.
For example, you didn't explain how cpu intensive your web apps are or how much load balancing (based on the front end API management and back end messaging) you expect you will need. While load balancers may not seem relevant to your Docker/K8s design question, the impact this can have on how you size your cluster nodes is obvious if you consider what impact too many nodes and a poorly distributed or overworked load balancer could have on your VLAN design.
The 'one app per VM' design you described suggests your app's workloads are pretty big (for an 8GB VM). Again need more info to delve into this more deeply, but assigning a DB VM per App VM seems a bit wasteful in terms of K8s worker/manager efficiency unless there's a severe reliability and resiliency requirement.
Next, what are your options to replace the magnetic HD with SSD? Costs for 100GB and 500GB SSD have dropped significantly and replacing your mag HD with SSD could certainly give your clusters a good deal of disk performance headroom when you scale up. In addition to performance, despite the added costs, overall system reliability (MTTF, MTTR) will increase dramatically. New SSD drives have 6GB data channels which is probably at least double what your mag HD have now. In building a Docker/K8s cluster of any size, disk channel bandwidth and processor core count can cover a multitude of design sins.
Finally, not sure what your project delivery timeframe is for this, but it might help your thinking all this over and making the best choices if you can take a step back and model this with SysML (using MS Visio) and look at your design and then apply Architectural Tradeoff Analysis Method.
ATAM and the theory behind it, despite its name, is really rather simple both in theory and in practice. ATAM gets its name because it not only
reveals how well an architecture satisfies particular quality goals (such as performance or "modifiability" or scaling in your case), but it also provides insight into how those goals interact with each other — how they
trade off against each other. Doing ATAM upfront does point out critical design decisions with the most far-reaching consequences and are the most difficult to change after a system has been implemented.
SysML and ATAM is also a kind of insurance policy against painting yourself into a corner when you are asked to cost justify your design choices or forecast TCO when a change in either the web app design or your VMWare infrastructure occurs in the future.
ATAM info here >>>
SysML info here >>>
