What is recommended to document an IT technology stack, including their relationship to one another, in a graph database?

Question

Working for a large company with over 500 IT staff and over a 1,000 servers, with each server running its own business applications, we have a tremendous information and co-ordination challenge in knowing which IT staff member to contact for which server. The problem of co-ordination is compounded with different IT staff being responsible for different layers of the IT stack. E.g., there are different teams who are responsible for hardware, virtualization, operating systems, application servers and the applications themselves.

Considering this challenge, within DevOps there is a requirement to define and document all the components that constitute the various technology stacks within an IT environment. Traditionally this might have been accomplished with a propriety CMDB solution.

I've investigated typical CMDB solutions such as BMC Atrium and others for this purpose, the problem however is that they stop at the level of documenting IT assets themselves, at a high level, as per the ITIL framework, but to do not address documentation of the IT Technology Stack in detail. I have also investigated tools such as Puppet, Ansible and Salt, but these tools focus more on software deployment and configuration, and not on the people co-ordination around software.

A workable solution for example, would define the various concepts, along with the key attributes important to these concepts:

• Hardware
• Virtualisation
• Operating Systems
• Application servers
• Applications

These concepts would then be associated with one another in terms of their relationships in order to form solutions. E.g., an application would depend on an application server, which would depend on an operating system, etc., Together this solution would be defined at the "Finance System". Having defined a system, all the metadata associated with these systems would be captured in order to facilitate co-ordination for each layer in the stack. I.e. the co-ordination of the technical support staff for each layer.

The purpose of such a solution would to do various queries against technology stacks, such as:

• To determine who supports which components
• Which components are out of date
• Which components need to be patched

With this in mind, what open source tools exist to define all the components of an IT technology stack, including their relationship to one another, in a graph database such as Neo4J?

Answer 1

Considering your first paragraph, the organization you're describing is a highly siloed org, which is exactly what a DevOps organization tend to avoid.

Considering this challenge, within DevOps there is a requirement to define and document all the components that constitute the various technology stacks within an IT environment. Traditionally this might have been accomplished with a propriety CMDB solution.

What you're describing here could be ITIL, which is a management system requiring documentation and you mix it with the fact a DevOps team will usually define the underlying layers as code, as such it gets back to any development documentation with the caveats of Code is Documentation often seen in Scrum methodology for an agile methodology of development (quick and short iterations aiming at minimal working solution at end of the iteration)

Disclaimer for the rest of this answer: I know more of chef and inspec and that's why I take them as exemple here; but they are not the only tools existing on the market, I won't open a debate on them as the better one is the one you're more comfortable with.

As such the rest of the question is a little biased and I, personally, didn't encounter an organization documenting the layer relation you describe more than the infrastructure as code and configuration management system code documentation. (Again, this doesn't mean no-one does it, I just never heard of it).
To illustrate from my company in a chef environment, an application cookbook will declare its dependencies (tomcat, jboss, nginx/php and on which OS, needed mount points for some shared datas and its DB schema name mostly) and expose its services URIs to be consumed by chef for other applications configuration, this sound like defining your 'Finance System' and the documentation for it is on this application cookbook README, with some more files if really needed.

Configuration management systems usually have a central reporting place, "chef-server" for datas an "manage UI" for presentation in chef world "ansible tower" for ansible world to name two of them, but they usually aim more at giving an oversight of the overall managed system than graphing the dependencies.

That said, for chef, the chef-server also act as a CMDB you can query with various tools (it return JSON datas from HTTP requests), the inter applications dependencies can be expressed in various way and there's no 'out of the box' method, each company will have its own way to declare them in the system for configuration purposes and as such you may leverage this to build your graph, but that's on your side.

In an infrastructure as code point of view, the infrastructure needs would be kept with the application, it is still the application who know what it needs under as middle-ware, which OS, with which locale, what are other services dependencies and what services this application offer).

Last thing I can think of if you want to manage those dependencies for documentation only are tools like glpi which is mainly a CMDB and a ticketing system, it takes advantage of documenting assets and their relation to be able to tell what is impacted when you open a ticket saying an application is down. coupled with ng-inventory it allows to query system states and as such can fulfill your query for patch needs, but in my opinion this is an audit system task, like could do inspect integrated within a chef run for exemple, as the next phase would be to fix the outdated systems by updating/patching them.