Handle OS and Software maintenance/updates on Hardware distributed to Customers

Question

In the last weeks I have tried to find a solution to the following scenario assuming that someone else had similar requirements and documented his solution but all my searches were unsuccessful. So now I'll just ask away and hope that someone with enough experience in this field can give me directions so that I don't take the wrong turn too many times.

The simplified scenario:

Let's say I have written a piece of software with a web interface and REST API for my customers which absolutely has to run in the local network at the customer's premises and will be provided to them as a "black box" hardware appliance. Apart from that I will essentially provide it "as a service" to them, meaning that maintenance and updates are my responsibility. I come to them with a small headless pre-installed and pre-configured PC, connect it to their network, perform only networking configuration and I'm done. The appliance is ready to communicate with any other software components running in the customer's network.

To be a bit more precise my software product runs under Linux (and also Windows), it should also be possible to build it as a Docker container (if that makes any sense for the proposed solution, although my know-how with Docker is still limited). I expect to set up CentOS 7 on the appliances.

Now I have two challenges:

• Keeping CentOS packages up-to-date
• ‎Keeping my software package up-to-date

To prevent all customers from experiencing the same bug at the same time I want to be in full control which customers update which packages at which point in time on the appliance (essentially a staged rollout). In regards to my own software package this could very well be in form of different update channels (beta, stable, etc.). Ideally I am also able to fetch and push configuration and other files from/to the appliance. It would be great if I could use existing solutions for everything instead of rolling my own mechanisms.

Also, the solution should scale to hundreds or even thousands of customers.

What I have thought of until now:

Spacewalk, Puppet, etc.

For managing the CentOS systems. However, I have no idea whether any one is good at managing machines that are not running inside a single company network.

RPM packaging my software component or My software as a Docker container

I don't know which one of these would be better regarding updates. Docker may be overkill, it's just a single software application on a single machine for each customer, I only thought that it would ease application updates. Maybe an RPM package would be preferable, that way I could handle update channels as different RPM repositories and control the updates themselves with Spacewalk/Puppet.

An OpenELEC/LibreELEC JeOS approach

I don't know what this is called exactly or whether there are pre-built solutions for this but I have first encountered it with OpenELEC. Essentialy the whole OS and application are packaged in a single non-modifiable and updatable (kernel) image which is installed on the target machine and there's a separate partition for user and local configuration data.

I know this is a pretty big question, but any pointers would be very appreciated.

Answer 1

Red Hat used to brand Spacewalk as "Red Hat Satellite 5" and decided to scrap it and build a new toolset from the ground up. Red Hat then based what they now brand as "Red Hat Satellite 6" on the Katello plugin for The Foreman.

In addition, the Foreman also has plugins for Docker, Puppet, and several other related systems.

But to be honest, the system is a bit rough and needs some polish. I frequently bump up against bugs and have to expend quite a bit of time making all the pieces work together properly - though it is still very cool and powerful.

You may, however, have an issue with any client networks that have strict security requirements. At my employer, we are not allowed to directly connect to the internet and are barely able to use proxies which require a rigorous security review and then we are still quite limited, so that might need to be a consideration for your updates model - You will be a no-go on those types of customers. On the other hand, that's part of the advantage of the "As-a-service" model. You don't have to worry about it or maintain it, so you will pick up more small-and-medium business as customers.

Answer 2

I can provide some personal experience here. I worked at a company that had the same need. The product was a BeagleBone (like an Arduino or Raspberry Pi) that ran our software and sent data back to our SaaS. It had to be installed inside customer networks, and be a little black box that they didn't touch.

When I walked into this company, management of the systems was a nightmare. Networking was a nightmare. OS updates, keeping them online, shipping new versions of the software, etc... it was a nightmare.

The path we ended up choosing ended up working really well: The entire application and supporting packages was shipped as a Docker container. That allowed us to easily ship application code changes and control versions of any supporting software (ffmpg, gcc, etc...).

With the application abstracted, that simply left making the OS as minimal as possible. All we needed was a dead simple OS that could run docker. One of our low level engineers actually made our own in-house fork of Debian (if I remember) and we shipped that embedded directly into the devices.

We already had a service contract with all of our customers to service these machines yearly, so when our field technician went on-site, the procedure was to replace the entire unit with a fresh hardware that came with the latest embedded OS. We didn't have to make many OS changes just to run Docker, so often times swapping wasn't even necessary. But out of procedure, we always swapped hardware whether it was an emergency service call or the yearly maintenance.

I can't say if this would be a standard approach or not, but just that it worked really well for us and was a life saver. It sounds like your idea of shipping the OS as an immutable image is along the same lines.

Edit: When I left, we were still using home grown bash scripts to automate the Docker container deployments, but the idea was to move toward a config management tool (Like Chef, Salt, or Ansible). The basic premise is that you have a private Docker registry somewhere, then on the devices you would simply run a couple commands:

docker pull my-private-registry/my-custom-image:latest
docker run my-custom-image:latest

If you are going to always be using the 'latest' tag, you could then have your devices ship with a cronjob that has then always shut down their local running Docker container, and re-pull the 'latest' tag every week during a scheduled maintenance window.