Case Study: Automating hardware-dependent tests for passport readers

By engaging with Cyberus to implement a test automation system, Veridos achieved:

• Cost Savings: Reduced hardware costs, by leveraging virtual machines
• Efficiency: Cut down testing time from hours to minutes
• Global Accessibility: Engineers can now run tests remotely and globally
• Reliability: Consistent and reproducible test results with CI/CD integration

According to Peter Zarnitz, Product Manager at Veridos: “Cyberus’ expertise helped us automate our system, run our tests effortlessly and save countless hours of development time. We are really happy that Cyberus engineers actively understand the problem and then propose, discuss and implement a high-quality, pragmatic solution in a very short time!”

When we began the project, Veridos engineers were relying on manual testing, which posed significant challenges due to regional regulations. Each development center worldwide had to deal with unique security features in identity documents, treated as highly confidential. Dummy documents, essential for testing these features, couldn’t be shared or moved outside their respective countries or development centers. This restriction forced engineers to request local tests for every software update, resulting in a cumbersome, time-consuming, and error-prone process. Clearly, the solution was to automate the testing process. But where should we start?

In less than 20 days, Cyberus’ test automation experts built an automation system that enables Veridos’ engineers to run the complete set of hardware-dependent tests without supervision. This system provides a proof of value and can be easily extended in the future. Learn more details in the next sections.

Test Automation System Design

To easily control the test automation remotely, the solution provides a REST API and a simple-to-use website. Using the website, the engineers can see which scanners are connected to the controller and configure them. The REST API provides CI/CD integration and can be used to schedule tests via scripts. Tests can also be scheduled manually. The systems allows Veridos engineers to run tests on scanners all over the world, which is important because the identity documents must not be moved from one location to another.

Another problem Veridos had was that they needed one computer for each scanner, which drives hardware costs significantly. To solve this problem, Cyberus leveraged virtual machines. When the controller receives a new request via its REST API, it starts a virtual machine, passes a scanner into the VM using USB-passthrough and then controls the VM via SSH. This has multiple advantages:

• multiple scanners can be connected to a single computer, saving hardware costs and space
• a base image can be cloned each time a VM is started, making sure every test starts from a clean state and results can be reproduced
• it is easy to remote control a VM and also to terminate a VM if a test hangs
• multiple tests can be run parallel on one computer

The software that controls the scanners is provided in an archive via the REST API. The controller copies the software into the VM, executes the software using a command that is also provided via the REST API, and then reports the result of the test back to the caller.

But what happens if a test fails and leaves a scanner in a weird state that prevents it from running new tests? Those cases can often be solved by power cycling the device, which is harder when a device may be anywhere in the world and the user has no physical access. To circumvent situations where a scanner isn’t usable for a longer period Cyberus power cycles each scanner after each test using a remote controllable power socket. When the controller starts it detects which scanner is connected to which port on the power socket and then only powers on a scanner when it is used.

All of this functionality is put into Python code and the function to run a test can be outlined as follows:

# The controller uses flask for the simple website and the REST API
@app.route("/run/<device_name>", methods=["POST"])
def run(device_name: str) -> flask.Response:
  test_config = flask.requests.files["test_config"]
  test_archive = flask.requests.files["test_archive"]

  # The `devices` object has a list of all connected scanners
  device = devices.find_device_by_name(device_name)

  device.lock() # The device is locked using a mutex
  device.power_on() # We power on the device at the power socket

  # The vm_manager creates a new VM for each test
  with vm_manager.get_vm(device_name) as vm:
    vm.start()

    # We wait until we can communicate with the VM
    conn = ssh.establish_connection_to(vm)
    vm.attach_device(device) # We pass the scanner into the VM
    # We transfer the archive into the guest and extract it
    conn.transfer_to_guest(test_archive)
    # We execute the given command and wait
    conn.execute(test_config.get("command"))
    # The test will create some result files which we compress
    # and transfer to the host
    results = conn.transfer_to_host(test_config.get("result_directory"))
    # We shutdown the VM
    vm.shutdown()
    vm.wait_for_shutdown()
  # Now we can power off and unlock the device
  device.power_off()
  device.unlock()
  # And return the results to the caller
  return flask.send_files(results, download_name="results.zip")

Of course there is error handling in the production code, but this is just to outline the steps of one test.

Summary and Outlook

The basic automation infrastructure allows for automatic testing on up to 3 devices at a single site. Tests are scheduled by a CI/CD pipeline step and run in parallel. The system provides a number of improvements for Veridos:

• Tests on multiple sites with multiple regional documents in parallel
• Tests are run for all changes of the software automatically
• Tests are run for all documents and without user intervention
• There is an automated paper trail of what tests have been executed

Ready to transform your testing process and adopt a Shift-Left Testing approach? Schedule a test automation consultation via our contact form today!