Security updates every night

Problem

Twelve web servers want their security updates, ideally every night. By hand that means: somebody does it during the day, at a risk to the visitors, or at night, and stays up for it. Add the exceptions that live only in a colleague's head: on web07 two PHP packages must not be touched, since a business application stopped answering after the update in March.

Wanted: updates every night without a person, the exceptions kept, and in the morning one look that is enough.

Solution

One schedule over the whole group, set up once. Nobody watches, nobody is woken up. What Heinzel Pro may not decide alone it leaves alone and reports; in the morning there is a mail.

Step 1: You set up the schedule

Input: Install the pending security updates on every server and check afterwards that all services run. What a server's journal exempts, leave alone and report as a finding.

Once, during the day. Name, server group, time, the task in your words, the level "Changes alone" and where the report goes. Nothing more.

Step 2: At three Heinzel Pro reads on all twelve

The schedule opens one session over the whole group. Heinzel Pro checks on every server what is pending and reads the server's journal.

On web07 a colleague held back two PHP packages in March because a business application stopped answering after the update. That is in the journal, so Heinzel Pro keeps to it.

Step 3: It installs what it may, and stops where the journal says so

On all twelve servers the updates run through on their own; on web07 the two held packages stay. Then Heinzel Pro checks whether a service failed anywhere.

What stays behind becomes a finding on the list, not a mail at three.

Step 4: In the morning you read the report

A mail shortly after three, read at eight: twelve servers updated, all services run, one finding on web07. What happens with it is the recipe Fixing a finding.