9.5. Benchmarking Workflow

This tutorial covers the complete benchmarking workflow in the dAIEDGE Middleware, from registering a benchmark request to setting and retrieving encrypted results.

Overview

The benchmarking system enables:

Requesting AI model benchmarks on specific hardware
Assigning hardware to benchmark tasks
Executing benchmarks with cryptographic verification
Storing encrypted results on blockchain
Verifying benchmark authenticity

Benchmark Lifecycle

1. Register Benchmark Request
   ↓
2. Assign Hardware to Benchmark
   ↓
3. Execute Benchmark (off-chain)
   ↓
4. Set Encrypted Result
   ↓
5. Retrieve and Decrypt Result

Prerequisites

Completed Hardware Registration, Dataset Registration, and AI Model Registration tutorials
Understanding of Role Management - requires BENCHMARKING_REQUESTER and BENCHMARKING_PRODUCER roles
Registered hardware, dataset (optional), and AI model (optional)

Benchmarking Roles

Required Roles

Role	Purpose	Who Needs It	Can Be Granted By
`BENCHMARKING_REQUESTER`	Request benchmarks, view results	Users requesting benchmarks	ADMIN_ROLE
`BENCHMARKING_PRODUCER`	Execute benchmarks, set results	Hardware owners executing benchmarks	ADMIN_ROLE
`HARDWARE_OWNER_ROLE`	Own hardware	Required to own hardware used in benchmarks	ADMIN_ROLE

Authorization Patterns

To register a benchmark request (Step 1):

Your identity must have BENCHMARKING_REQUESTER role

To assign hardware to benchmark (Step 2):

Your identity must have BENCHMARKING_PRODUCER role AND
You must own the hardware being assigned (checked via HardwareRegistry)

To set benchmark results (Step 4):

Your identity must have BENCHMARKING_PRODUCER role AND
You must own the hardware that was assigned to the benchmark

To retrieve benchmark results (Step 5):

You must be the benchmark requester (owner of the benchmark)
Requires wallet password for result decryption

Role Dependencies

The BENCHMARKING_PRODUCER role requires hardware ownership:

BENCHMARKING_PRODUCER (role)
         +
HARDWARE_OWNER_ROLE (role) + Hardware Ownership (resource)
         ↓
Can execute benchmarks on owned hardware

Important: Simply having BENCHMARKING_PRODUCER role is not sufficient. You must also:

Have HARDWARE_OWNER_ROLE
Own (have registered) the hardware you want to use for benchmarks

This is enforced by the smart contract using Pattern 3: Role AND Ownership (see Role Management).

Role Assignment

See Role Management tutorial for granting roles.

Step 1: Register Benchmark Request

Create a new benchmark task.

Endpoint

POST /api/v1/benchmark

Request

curl -X POST https://middleware-daiedge.bisite.usal.es/api/v1/benchmark \
  -H "Content-Type: application/json" \
  -H "x-session-id: {SESSION_ID}" \
  -d '{
    "walletId": "000007e2a9f5c1b8d4e7a3f6",
    "password": "WalletPass123!",
    "did": "did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
    "modelHash": "0xc4e7a9d2f5b8c1e4a7d9f2b5c8e1a4d7f9c2b5e8a1d4f7c9e2b5a8d1f4e7c9e2",
    "datasetHash": "0x5a8d1f4c7e9b2a5d8f1c4a7e9d2f5b8c1e4a7d9f2c5b8e1a4d7f9c2e5a8d1f4c",
    "benchmarkType": "TYPE1",
    "target": "GPU+CUDA",
    "runtime": "TensorFlow Lite"
  }'

async with daiedge_middleware_client.ApiClient() as api_client:
    benchmark_api = daiedge_middleware_client.RegisterBenchmarkApi(api_client)
    result = await benchmark_api.benchmark_post(
        daiedge_middleware_client.RegisterBenchmarkBody(
            wallet_id="000007e2a9f5c1b8d4e7a3f6",
            password="WalletPass123!",
            did="did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
            model_hash="0xc4e7a9d2f5b8c1e4a7d9f2b5c8e1a4d7f9c2b5e8a1d4f7c9e2b5a8d1f4e7c9e2",  # pass "" if not applicable
            dataset_hash="0x5a8d1f4c7e9b2a5d8f1c4a7e9d2f5b8c1e4a7d9f2c5b8e1a4d7f9c2e5a8d1f4c",  # pass "" if not applicable
            benchmark_type="TYPE1",
            target="GPU+CUDA",
            runtime="TensorFlow Lite",
        )
    )
    benchmark_id = result.id
    print(f"Benchmark registered: id={benchmark_id}")

Request Body

Field	Type	Required	Description
`walletId`	string	Yes	Your wallet ID
`password`	string	Yes	Wallet password
`did`	string	Yes	Your identity DID (format: `did:ethr:{network}:0x...`, must have `BENCHMARKING_REQUESTER` role)
`modelHash`	string	Yes	AI model ID (max 255 chars, pass `""` if not applicable)
`datasetHash`	string	Yes	Dataset ID (max 255 chars, pass `""` if not applicable)
`benchmarkType`	string	Yes	Benchmark type: `TYPE1`, `TYPE2`, or `TYPE3`
`target`	string	Yes	Target platform (max 255 chars)
`runtime`	string	Yes	ML runtime name (max 255 chars)

Benchmark Types

Type	Description
`TYPE1`	Inference benchmark
`TYPE2`	Training benchmark
`TYPE3`	Custom benchmark

Response (200 OK)

{
  "id": "0x7e9c2f5a8d1b4e7c9a2f5d8b1c4e7a9d2f5b8c1e4a7d9f2c5b8e1a4d7f9c2e5a"
}

Returns the benchmark ID.

Error Responses

Code	Error	Cause
404	`WALLET_NOT_FOUND`	Wallet doesn’t exist
400	`WRONG_PASSWORD`	Wallet password incorrect
400	`INVALID_DID`	Identity format invalid
400	`INVALID_MODEL_HASH`	Model hash > 255 chars
400	`INVALID_DATASET_HASH`	Dataset hash > 255 chars
400	`INVALID_BENCHMARK_TYPE`	Type not TYPE1, TYPE2, or TYPE3
400	`INVALID_TARGET`	Target empty or > 255 chars
400	`INVALID_RUNTIME`	Runtime empty or > 255 chars
403	`INVALID_WALLET`	Wallet doesn’t belong to you
500	`TRANSACTION_ERROR`	Blockchain transaction failed (check `BENCHMARKING_REQUESTER` role)

Step 2: Assign Hardware to Benchmark

Assign a hardware device to execute the benchmark.

Endpoint

POST /api/v1/benchmark/{id}/set-hardware

Request

curl -X POST https://middleware-daiedge.bisite.usal.es/api/v1/benchmark/{BENCHMARK_ID}/set-hardware \
  -H "Content-Type: application/json" \
  -H "x-session-id: {SESSION_ID}" \
  -d '{
    "walletId": "000008f3b7a6d2c9e5f1a8b4",
    "password": "WalletPass123!",
    "did": "did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
    "deviceId": "0x2d5f8a1c4e7b9d2f5a8c1e4d7b9f2a5c8e1d4f7a9c2e5f8b1d4e7a9c2f5e8d1f"
  }'

async with daiedge_middleware_client.ApiClient() as api_client:
    benchmark_api = daiedge_middleware_client.RegisterBenchmarkApi(api_client)
    await benchmark_api.benchmark_id_set_hardware_post(
        id=benchmark_id,
        request=daiedge_middleware_client.SetHardwareBody(
            wallet_id="000008f3b7a6d2c9e5f1a8b4",
            password="WalletPass123!",
            did="did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
            device_id="0x2d5f8a1c4e7b9d2f5a8c1e4d7b9f2a5c8e1d4f7a9c2e5f8b1d4e7a9c2f5e8d1f",
        ),
    )
    print(f"Hardware assigned to benchmark {benchmark_id}")

Request Body

Field	Type	Required	Description
`walletId`	string	Yes	Your wallet ID
`password`	string	Yes	Wallet password
`did`	string	Yes	Your identity DID (format: `did:ethr:{network}:0x...`, must have `BENCHMARKING_PRODUCER` role and own hardware)
`deviceId`	string	Yes	Hardware ID from registration

Response (200 OK)

{}

Error Responses

Code	Error	Cause
404	`WALLET_NOT_FOUND`	Wallet doesn’t exist
404	`BENCHMARK_NOT_FOUND`	Benchmark ID doesn’t exist
404	`HARDWARE_NOT_FOUND`	Hardware ID doesn’t exist
400	`WRONG_PASSWORD`	Wallet password incorrect
400	`INVALID_DID`	Identity format invalid
403	`INVALID_WALLET`	Wallet doesn’t belong to you
500	`TRANSACTION_ERROR`	Blockchain transaction failed (check roles and hardware ownership)

Authorization

You can assign hardware if:

You have BENCHMARKING_PRODUCER role, AND
You have HARDWARE_OWNER_ROLE, AND
You own the hardware (registered it in HardwareRegistry)

This uses Authorization Pattern 3: Role AND Ownership from the Role Management tutorial.

License Requirements

If the hardware has a policy defined (see Policy Management), the user requesting the benchmark must have a valid license for that hardware.

The license is checked and consumed automatically when you call set-hardware.
If the user does not have a valid license, the transaction will revert with “No valid license”.
See Licensing for instructions on how to request and obtain a license.

Step 3: Execute Benchmark (Off-Chain)

This step happens off-chain on the actual hardware. Your benchmarking infrastructure should:

Monitor blockchain for benchmark assignments to your hardware
Download model and dataset from URIs
Execute benchmark on hardware
Collect results (accuracy, latency, throughput, etc.)
Prepare encrypted result report

Note: This step is implementation-specific and not covered by the API directly.

Step 4: Set Benchmark Result

Upload the encrypted benchmark result to the blockchain.

Endpoint

POST /api/v1/benchmark/{id}/set-result

Request

curl -X POST https://middleware-daiedge.bisite.usal.es/api/v1/benchmark/{BENCHMARK_ID}/set-result \
  -H "Content-Type: application/json" \
  -H "x-session-id: {SESSION_ID}" \
  -d '{
    "walletId": "000009a4c8e5b2d7f9a3c6e1",
    "password": "WalletPass123!",
    "did": "did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
    "report": "Inference time: 45ms, Accuracy: 94.2%, Memory: 512MB",
    "result": true
  }'

async with daiedge_middleware_client.ApiClient() as api_client:
    benchmark_api = daiedge_middleware_client.RegisterBenchmarkApi(api_client)
    await benchmark_api.benchmark_id_set_result_post(
        id=benchmark_id,
        request=daiedge_middleware_client.SetBenchmarkResultBody(
            wallet_id="000009a4c8e5b2d7f9a3c6e1",
            password="WalletPass123!",
            did="did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
            report={"report": "Inference time: 45ms, Accuracy: 94.2%, Memory: 512MB"},
            result=True,
        ),
    )
    print("Result set successfully")

Request Body

Field	Type	Required	Description
`walletId`	string	Yes	Your wallet ID
`password`	string	Yes	Wallet password
`did`	string	Yes	Your identity DID (format: `did:ethr:{network}:0x...`, must have `BENCHMARKING_PRODUCER` role and own hardware)
`report`	string	Yes	Benchmark report string (max 1000 chars)
`result`	boolean	Yes	Success status: `true` or `false`

Report Field

The report field can contain:

Performance metrics (latency, throughput, accuracy)
Resource utilization (memory, CPU, GPU)
Test results
Error messages (if result is false)

Maximum length: 1000 characters

Disclaimer: The actual information stored from the benchmarking result is a work in progress and subject to change.

Result Field

true: Benchmark succeeded
false: Benchmark failed

Response (200 OK)

{}

Encryption

The backend automatically encrypts the result using:

ECDH shared secret between user and operator
AES-256-GCM encryption
Stored on blockchain: ciphertext, IV, auth tag

Error Responses

Code	Error	Cause
404	`WALLET_NOT_FOUND`	Wallet doesn’t exist
404	`BENCHMARK_NOT_FOUND`	Benchmark ID doesn’t exist
400	`WRONG_PASSWORD`	Wallet password incorrect
400	`INVALID_DID`	Identity format invalid
400	`REPORT_TOO_LONG`	Report > 1000 chars
403	`INVALID_WALLET`	Wallet doesn’t belong to you
500	`ENCRYPT_ERROR`	Result encryption failed
500	`TRANSACTION_ERROR`	Blockchain transaction failed

Step 5: Retrieve Benchmark Result

Get the benchmark result and decrypt it.

Endpoint

GET /api/v1/benchmark/{id}?wallet={walletId}&password={password}&userDID={userDID}

Request

curl -X GET "https://middleware-daiedge.bisite.usal.es/api/v1/benchmark/{BENCHMARK_ID}?wallet=00000a5d9f6c3b8e1a4d7f2c&password=WalletPass123%21&userDID=did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7" \
  -H "x-session-id: {SESSION_ID}"

async with daiedge_middleware_client.ApiClient() as api_client:
    benchmark_api = daiedge_middleware_client.RegisterBenchmarkApi(api_client)
    benchmark = await benchmark_api.benchmark_id_get(
        id=benchmark_id,
        wallet="00000a5d9f6c3b8e1a4d7f2c",
        password="WalletPass123!",
        user_did="did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
    )
    print(f"target: {benchmark.target}")
    print(f"runtime: {benchmark.runtime}")
    print(f"approved: {benchmark.approved}")

Query Parameters

Parameter	Type	Required	Description
`wallet`	string	Yes	Wallet ID for decryption
`password`	string	Yes	Wallet password (URL encode special chars, e.g. `!` → `%21`)
`userDID`	string	Yes	Your DID (must have `BENCHMARKING_PRODUCER` role)

Response (200 OK)

{
  "modelHash": "0x9f2c5a8d1e4b7f9c2a5d8e1f4c7a9d2f5b8c1e4a7d9f2c5b8e1a4d7f9c2e5a8d",
  "datasetHash": "0x3a7d9f2c5b8e1a4d7f9c2e5a8d1f4c7e9b2d5f8a1c4e7b9d2f5a8c1e4f7b9d2e",
  "target": "GPU+CUDA",
  "runtime": "TensorFlow Lite",
  "deviceId": "0x6c9e2a5f8d1b4c7e9a2d5f8b1c4e7a9d2f5b8c1e4a7d9f2c5b8e1a4d7f9c2e5a",
  "approved": true,
  "userDID": "0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
  "created": 1697000000000
}

Decryption

The backend:

Retrieves encrypted data from blockchain
Derives shared secret using your wallet’s private key
Decrypts result using AES-256-GCM

Error Responses

Code	Error	Cause
404	`BENCHMARK_NOT_FOUND`	Benchmark ID doesn’t exist
404	`WALLET_NOT_FOUND`	Wallet doesn’t exist
400	`WRONG_PASSWORD`	Wallet password incorrect
500	`TRANSACTION_ERROR`	Blockchain query failed
500	`ENCRYPT_ERROR`	Decryption failed

Step 6: List Benchmarks

Browse all benchmarks on the platform.

Endpoint

GET /api/v1/benchmark?userDID={userDID}&page={page}

Request

curl -X GET "https://middleware-daiedge.bisite.usal.es/api/v1/benchmark?userDID=did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7&page=1" \
  -H "x-session-id: {SESSION_ID}"

async with daiedge_middleware_client.ApiClient() as api_client:
    benchmark_api = daiedge_middleware_client.RegisterBenchmarkApi(api_client)
    benchmarks = await benchmark_api.benchmark_get(
        user_did="did:ethr:development:0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7",
        page=1,
    )
    print(f"total: {benchmarks.total}")
    for b in benchmarks.benchmarks:
        print(f"  target={b.target} runtime={b.runtime} device={b.device_id}")

Response (200 OK)

{
  "benchmarks": [
    {
      "id": "0x8d1f4a7c9e2b5d8f1a4c7e9b2d5f8a1c4e7b9d2f5a8c1e4f7b9d2e5a8c1f4e7b",
      "modelHash": "0xa5c8e1d4f7b9c2e5a8d1f4c7e9b2d5f8a1c4e7b9d2f5a8c1e4f7b9d2e5a8c1f4",
      "datasetHash": "0x2f5a8c1d4e7b9f2a5c8d1e4f7b9c2e5a8d1f4c7e9b2d5f8a1c4e7b9d2f5a8c1e",
      "deviceId": "0x7b9d2e5f8a1c4d7b9e2f5a8c1d4e7b9c2f5a8d1e4f7b9c2e5a8d1f4e7b9c2e5a",
      "target": "GPU+CUDA",
      "runtime": "TensorFlow Lite",
      "created": 1697000000000,
      "userDID": "0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb7"
    }
  ],
  "page": 1,
  "totalPages": 15,
  "total": 368
}

Note: This endpoint requires BENCHMARKING_PRODUCER_ROLE and the userDID query parameter (not did).

Next Steps

Marketplace - Trade benchmarking services
Rewards and Tokens - Earn rewards for executing benchmarks

9.5. Benchmarking Workflow

Overview

Benchmark Lifecycle

Prerequisites

Benchmarking Roles

Required Roles

Authorization Patterns

Role Dependencies

Role Assignment

Step 1: Register Benchmark Request

Endpoint

Request

Request Body

Benchmark Types

Response (200 OK)

Error Responses

Step 2: Assign Hardware to Benchmark

Endpoint

Request

Request Body

Response (200 OK)

Error Responses

Authorization

License Requirements

Step 3: Execute Benchmark (Off-Chain)

Step 4: Set Benchmark Result

Endpoint

Request

Request Body

Report Field

Result Field

Response (200 OK)

Encryption

Error Responses

Step 5: Retrieve Benchmark Result

Endpoint

Request

Query Parameters

Response (200 OK)

Decryption

Error Responses

Step 6: List Benchmarks

Endpoint

Request

Response (200 OK)

Next Steps

Related Topics