Test Runner API (Deprecated)

Table of Contents

Use this API to run a regression test that leverages the previous output of flows that were successfully run and compares the representative ground truth (also known as golden output) to the test output.

See Instabase API authorization and response conventions for authorization, success response, and error response convention details.

url_base for instabase.com is https://www.instabase.com/api/v1
For an on-premises installation of Instabase, your url_base will be different.

Run a test

The Test Runner API provides two different ways to use Test Runner:

Test Runner with Test-Suite
Test Runner with Test-Suite path

The Get test execution status API provides the execution status of a job.

Test Runner with Test-Suite

Use this API to run a test by encapsulating the entire test-suite in the request body.

Request

Send a POST request to url_base/ibtest/run_test_async with the post body encoded as JSON:

import json, requests

url_base = 'https://localhost'
url = url_base + '/api/v1/ibtest/run_test_async'
url = url_base + '/ibtest/run_test_async'
token = 'your token here'

ibtest_content =  {
 "version" : "1",
 "tests" : [
   {
       "name":"run-flow-test",
       "api":"run_flow_async",
       "payload": {
         "ibflow_path": "prai/my-repo/fs/Instabase Drive/flow-experiment/workflows/sample-3.ibflow",
         "input_folder": "input",
         "compile_and_run_as_binary": True,
         "delete_out_dir":True,
         "output_has_run_id":False,
         "skip_steps":[],
         "enable_ibdoc": True,
         "log_to_timeline": False
       },
       "comparison_list": [
         {
           "comparator": "ibocr_comparator",
           "params": {
             "test_output" : [
             "s4_merge_files/out.ibocr"
           ],
             "golden_output": [
             "prai/my-repo/fs/Instabase Drive/flow-experiment/samples/sample-3/out-10-02-2020/sample-3/out/s4_merge_files/out.ibocr"
           ]
           }
         }
       ]
   }
 ]
}

api_args = {
 'ibtest_content': json.dumps(ibtest_content)
}
json_data = json.dumps(api_args)

headers = {
 'Authorization': 'Bearer {0}'.format(token),
 'Instabase-API-Args': json.dumps(api_args)
}

r = requests.post(url, data=json_data, headers=headers)
resp_data = json.loads(r.content)
print(resp_data['job_id'])

The body of the request must include all of the test specification fields defined in the .ibtest file.

Test Runner with Test-Suite path

Use this API to run a test that is defined by an .ibtest file.

Request

Send a POST request to url_base/ibtest/run_test_runner_async with the post body encoded as JSON:

import json, requests

url_base = 'https://localhost'
url = url_base + '/api/v1/ibtest/run_test_runner_async'
url = url_base + '/ibtest/run_test_runner_async'
token = 'your token here'

api_args = {
 'ibtest_path': '<path to your ibtest file on Instabase>',
 'output_folder': '<path to the folder where you want to store the result logs of the test>'
}

json_data = json.dumps(api_args)

headers = {
 'Authorization': 'Bearer {0}'.format(token),
 'Instabase-API-Args': json.dumps(api_args)
}

r = requests.post(url, data=json_data, headers=headers)
resp_data = json.loads(r.content)
print(resp_data['job_id'])

Response

The HTTP call returns immediately while the binary execution proceeds asynchronously in the background.

For the Test Runner with Test-Suite path API, the result is stored in the output_folder when the state of the job is ‘DONE’.

Get test execution status

Use this API to get the execution status of a job.

Request

Send a GET request to url_base/jobs/status?job_id={job_id}:

GET url_base/jobs/status?job_id={job_id}

Response

The response body is a JSON object reporting the status of the execution:

  HTTP STATUS CODE 200

  # body
  {
    "status": "OK",
    "state": <string: one of a number of states>,
    "job_id": <string>
  }

status: "OK" or "ERROR"
state: "PENDING" or "DONE"
job_id: The unique identifier for the job.