TL;DR: Measures of size can be split roughly in three different categories that I would define as depth in what you
manage vs. what you outsource or consume as a service, breadth of
services supported and height in numbers of instances,servers and
customers. Measures of complexity largely depend on selected
systems architecture, organizational structure of people supporting it and skill sets required. Where depth and skill sets
go hand in hand adding both to size and complexity.
Note: Most of the following size requirements can also add to complexity through systems architecture, skill sets and organization structure needs.
Depth of Infrastructure
How much of the infrastructure you are outsourcing to others in increased levels of depth:
- Do you simple use Software as a Service for all that you do?
- Do you operate fully in public cloud, private or hybrid cloud or
using some PaaS?
- Do you use Infrastructure as a Service?
- Do you use Hosted and Managed Infrastructure in rented DC space?
- Do you own or rent the hardware?
- Does the provider manage infrastructure monitoring?
- Does the provider manage basic system administration?
- Does the provider manage hardware failures and maintenance?
- Does the provider manage racking and server installation?
- Does the provider manage internal networking?
- Does the provider manage internet connectivity and routing?
- Do you have just data centers with contracted remote hands?
- Do you host everything on premises or in your own datacenters?
Breadth of Infrastructure
What are the different types of services you support?
- Compute Resources
- Bare metal servers
- Virtualization layer (VMWare)
- Container layer (docker, k8, mesos)
- Serverless layer (lambda, functions)
- Storage Resources
- Standalone storage appliances
- RAID in servers
- Large standalone relational database clusters
- Time series databases
- Object store clusters
- Network Resources
- Observability Resources
- System Log servers
- Metrics and Graph systems
- Search clusters
- Automation Resources
- Backup/Recovery Resources
- Complex compounded services
Height of Infrastructure
- What is the scale for each resources that you need? Do you operate the services on single server/instance or do you need use clusters of machines?
- What is the level of redundancy you need?
- What are your availability requirements?
- What are your requirements for latency and throughput of services?
- Do you need geographically distributed infrastructure? (international business, latency requirements or regulation compliance like GDPR, data localization laws, etc)
- Do you need multiple datacenters in each geography?
Just very briefly...
When it comes to Infrastructure complexity, it quite closely follows the complexity of the distributed systems supported by the infrastructure. You have to take into account two types of systems:
- Distributed systems supporting individual services.
- Distributed system created by interdependencies of the services.
Distributed Systems Complexity
Every service your infrastructure supports can have in and of itself a different level of complexity with varying levels of requirements on the infrastructure. Systems supporting services can range through:
- Single threaded.
- Multi-threaded (shared memory, shared disk)
- Parallel systems with data sharding
- HA Failover (Primary / Standby) (Cold, Warm, Hot)
- HA Cluster (N+M)
- Real-time Clusters
Interdependence of Services
Let me start with an example. Let's say your infrastructure reports test results into ElasticSearch cluster. Your pager depends on monitoring and test data provided by ElasticSearch. Geographical distribution of ElasticSearch cluster makes it depend on your datacenter networking connectivity. Now one of your internet providers decided to make an unannounced maintenance on Saturday night, throughput drops, your traffic is rerouted to backup provider, monitoring traffic is de-prioritized to customer data traffic, ingest of monitoring events slows and your pager goes crazy.
Every time two services, two parts of the infrastructure depend on each other, they create a new single distributed system, whose complexity should be judged independently. Such dependency can be either removed or reduced. Remember that system is only as redundant and as available as the intersection of all services that it depends on.
Further examples of complexity increasing factors:
- Dependencies on external services.
- Attempts to mitigate failure from service dependency.
- Multiple providers
- Caching of data
This is a chapter in and of itself ... people are often overlooked part of the overall system of IT infrastructure. We rarely think about redundancy, availability and latency factors when it comes to people, but just as with computers, these same issues affect organizations maintaining infrastructure and the complexity of it can sometimes easily outweigh the complexity of the computer systems. People involved in maintaining your infrastructure can span multiple time zones, languages, geographic locations, companies, pay scales and legal codes. Any of those factors are signs of increase in complexity.