What the client signs when an ERP system is accepted

An ERP system is usually accepted by a signature under a list of functions. Each function was shown and each one worked. Months later the same operations take noticeably longer, and nothing in the signed document says how fast they should be.

The acceptance checked what the system does, on a small database and with one user at a time. What the client signs, what is automated and what is tested under load are decided long before the system goes live.

What the client signs

Two documents matter.

The test plan. For each function it states the input data and the expected result. It is part of the specification and is approved with it, so the developer, the tester and the user check the same thing. For the client's people it is often the most readable part of the specification: it describes what is entered and what comes out, not the algorithm.

The check list. A short list of the capabilities the system needs for the processes in scope. A line is marked only when its tests have passed, and the manager on the client side makes the mark or checks it personally.

What is worth automating

A system keeps changing after acceptance: errors are fixed, functions are added, updates are installed. A change in one place can break a function in another. So regression tests are run before every release to the working database: prepared scenarios that cover every function in use.

Running them by hand every time is expensive, so they are automated first, together with the smoke set, a minimal check for obvious failures. A rule of thumb from practice: an automated test takes two to three times the effort of one manual run and pays back when it is repeated four to ten times.

Rare checks stay manual. Automated tests need upkeep, which can cost more than manual testing, so not everything is automated.

Why an accepted system slows down

Inefficient code is hard to notice on a small sample of data, in 1C:ERP as in any ERP system. A few months of trial operation do not reveal it either: too little data has accumulated by then. The problem appears later, when the project is closed. Two things are tested to prevent this.

Volume. The test database is filled to the volume the working system will reach several months after the start. Usually such data has to be generated.

Concurrency. The important operations are run by many users at once, simulated or real. Mutual locks, timeouts and operations that hang do not show in a test with one user.

Speed as a number

"Slow" cannot be accepted or rejected. A list of key operations can. A key operation is a single user action that is critical for the business and performed by many users at once. For each one the client names a target time.

APDEX, an open index of response time, turns the measurements into one figure. A result under the target time counts as one, a result under four times the target as one half, anything longer as zero, and the index is the average. By the usual reading, a value from 0.85 is good and from 0.94 excellent.

Six questions before the signature

• Is there a test plan with input data and expected results, approved as part of the specification?
• Who on the client side confirms each line of the check list?
• Which tests are repeated before every release, and which are automated?
• Is there a list of key operations with a target time for each?
• How much data was in the test database, and how many simultaneous users in the load test?
• Does the client see the defect register, and does it separate what the supplier's team found from what users reported?

An open question is cheaper to settle before the signature than after it.