Application checks after a restore

The four checks the Sefthy Agent runs inside the restored machine, how to write them, how to try them straight away against the production server and where the outcomes show up.

The restore test verifies that the machine boots. Whether the ERP answers is another question, and the person who knows the answer is the one who installed that ERP. So you write the application checks, and the Sefthy Agent, the program that lives on the server, runs them inside the restored copy.

Adding an application check and trying it against the production server

Where you write them

  1. Open the DR page and click the three dots of More actions, at the top right.
  2. Under Configuration click Post-restore validation. Next to the entry you read how many checks you have already defined, or not configured in amber.
  3. In the window click Add check and pick what you want to check among the four types.
  4. Write the Check name, fill in the other fields and click Add. The check appears under Defined checks, and from there you edit or delete it.

You can define up to ten per DR. On the eleventh the Console refuses and asks you to delete one first.

The four types

Path or file. The file or folder must exist. On Windows write a path with a drive letter, for example D:\SQLDATA\acme.mdf, or a UNC path in the form \\server\share. On Linux an absolute path, for example /var/lib/mysql/acme/acme.ibd. The Console knows the operating system of the server and rejects the wrong syntax when you save.

TCP port. The port must accept connections. Host is already localhost and usually stays that way, Port is the number, for example 1433 for SQL Server or 5432 for PostgreSQL, and Timeout (s) starts at 10 and goes up to 30.

Service. The service must be running. Write the Windows service name or the systemd unit, not the display name, so MSSQLSERVER and not SQL Server, nginx and not Nginx web server.

Web address. The address must answer. In Address put the URL, for example http://localhost/erp/login, in Expected status the HTTP code you expect, 200 by default, and in Expected text a string that must appear in the page, an optional field. Only http and https are allowed and the host must be the protected machine, so localhost, 127.0.0.1 or an IP address on record for this DR. A check must never be able to become a probe pointed at somebody else.

Trying them on the production server

A badly written check surfaces at next week’s test, when it is no longer any use. Try now runs it immediately, on the real server, with the same engine that will run inside the restored copy.

  1. In the window click Try now.
  2. You get Checks running on the server…. A few dozen seconds before the outcomes show up is normal.
  3. Next to each check the outcome appears, passed, failed or not run, with the detail reported by the server and how long it took.

If the server does not answer you read No answer from the agent, and pressing Try now again once the server is online is enough. If the server is unreachable right then the run stays queued and starts as soon as it is back. If the DR has no agent at all the run does not start and the Console tells you.

The not run result is not a failure. It means the check was not executed, or that the answer was not readable, and when in doubt the Console never calls it passed.

Where the outcomes show up

In the automatic restore test report, under the three outcome tiles, one row per check with its detail. The verdict on the checks stays separate from the one on boot and network, so a failed check does not turn a successful restore into a failure. How to read the report is in The automatic restore test and its report.

If outcomes arrive for only some of the checks, or do not arrive at all, the report marks them partial in amber rather than failed. The data is missing, it is not a negative verdict.

The same four types come back in the Orchestration of DR as the condition to move to the next step, with the Agent prefix in Check Type. Start the database, wait for 1433 to answer, then start the web server. The full round is in Orchestration of DR.

When changes take effect

From the next test. When a test starts the Console freezes the checks as they are at that moment, so a check added afterwards does not rewrite yesterday’s report and a deleted one stays in the old ones. The window reminds you at the bottom with Changes apply from the next test.

In the press