Test Custom Capabilities

Use the Smart Home AI Toolkit to test your custom capabilities at two scopes:

• Capability – Test a single custom capability in isolation to validate that Alexa correctly understands the directives, properties, and parameters defined in the interface.
• Device – Test the complete set of custom and pre-built capabilities included in your device to validate that Alexa correctly identifies the capability to invoke across all included interfaces.

Your custom capability and device must each pass evaluations before you can publish for review. If the evaluation fails, the Alexa+ smart home agent gives you actionable feedback pointing to the capability, directive, or parameter description to fix. If you make a change to the capability, the toolkit marks the evaluations as out of date (Stale) and you must rerun the evaluations.

Generate an utterance dataset

Use the Smart Home AI Toolkit to generate an utterance dataset that maps natural language utterances to the directives defined in your custom capability or device. The generated dataset represents how customers would naturally speak to control your device.

After the utterance generation completes, you can download the generated dataset as a JSON file in the following format:

[{
    "dialogId": "d0c0dce2837d494db0a8f69454980972",
    "turns": [{
        "utterance": "feed my dog",
        "actionPlans": [{
            "apiName": "Alexa.Custom.<capabilityName>",
            "parameters": {}
        }]
    }]
}]

Generate utterances for a capability

You can generate utterances for a specific capability in the capability workspace.

To generate test utterances for a single capability

1. Navigate to Capabilities > your capability name, and then select the Test tab.
2. To generate the utterance dataset, expand Generate Test Utterances, and then choose Generate.
3. If the generation fails, fix any errors in the capability definition.
   Tip: To fix errors, chat with the agent to add, update, or remove directives, properties, or responses.
4. To regenerate the utterance dataset, expand Generate Test Utterances, and then choose Rerun.
5. After the dataset generation completes successfully, expand Generate Test Utterances to view the generated utterances. To download the utterances, choose Download dataset.
6. Review the generated utterances. If any mappings don't reflect how customers would naturally speak, instruct the agent to update the utterances or capability, and then choose Rerun.

Generate utterances for a device

You can generate a combined set of test utterances for all capabilities included in your device in the device workspace.

To generate test utterances

1. Navigate to Devices > your device name.
2. On the menu, choose Test, and then expand Full Device Test.
3. To generate the utterance dataset, expand Generate Test Utterances, and then choose Generate.
4. If the generation fails, fix any errors in the failed custom capability definition.
   Tip: To fix errors, chat with the agent to add, update, or remove directives, properties, or responses. Then regenerate the capability test utterances and run the batch evaluation.
5. To generate an utterance dataset for the device, under Generate Test Utterances, choose Rerun.
6. After the dataset generation completes, to view the generated utterances, expand Generate Test Utterances. To download the utterances, choose Download dataset.
7. Review the generated utterances. If any mappings don't reflect how customers would naturally speak, instruct the agent to update the utterances or capability, and then choose Rerun.

Run a batch evaluation

Batch evaluation validates that Alexa correctly understands your custom capability or the collection of capabilities included in your device. The evaluation tests the generated utterance dataset against Alexa's natural language understanding to confirm that each utterance resolves to the expected directive with the expected parameters.

Your custom capability and device must each pass batch evaluation before you can publish for review. If the evaluation fails, the agent gives you actionable feedback pointing to the exact directive or parameter description to fix. You can run the capability evaluation in the capability workspace or the device workspace.

Run a batch evaluation for a capability

For a single custom capability, you can run an utterance evaluation against the generated utterance dataset.

To run a batch evaluation for a capability

1. Navigate to Capabilities > your capability name, and then select the Test tab.
2. To test the utterance dataset, expand Batch Evaluation, and then choose Run Evaluation.
3. If the evaluation fails, fix any errors in the capability definition.
   Tip: To fix errors, chat with the agent to add, update, or remove directives, properties, or responses.
4. If you change the capability definition, regenerate the utterance dataset. Under Generate Test Utterances, choose Rerun.

Run a batch evaluation for a device

Using the generated device-scoped utterance dataset, you can run an utterance evaluation across all capabilities included in the device.

To run a batch evaluation for a device

1. Navigate to Devices > your device name.
2. On the menu, choose Test, and then expand Full Device Test.
3. To test the utterance dataset, expand Batch Evaluation, and then choose Run Evaluation.
4. If the evaluation fails, fix any errors in the capability definition.
   Tip: To fix errors, chat with the agent to add, update, or remove directives, properties, or responses.
5. If you change the capability definition, regenerate the utterance dataset. Under Generate Test Utterances, choose Rerun.

Test custom utterances

To test utterances not found in the generated dataset, such as edge cases and errors, you can create and test a set of custom utterances. Use custom utterance testing to validate specific phrases that customers might say, including edge cases that the generated dataset might not cover. Testing custom utterances is optional.

Test custom utterances for a capability

You can define a custom utterance set for a single capability.

To test custom utterances for a capability

1. Navigate to Capabilities > your capability name, and then select the Test tab.
2. Expand Custom Utterance Testing, and then enter one or more customer utterances, one per line.
3. To run the test, choose Run Custom Test.
4. If the test fails, fix any errors in the utterance or capability definition.
5. To rerun the utterances, choose Rerun.

Test custom utterances for a device

You can define a custom utterance set for all capabilities included in a device.

To test custom utterances for a device

1. Navigate to Devices > your device name.
2. On the menu, choose Test, and then expand Full Device Test.
3. Expand Custom Utterance Testing.
4. Enter one or more customer utterances, one per line.
5. To run the test, choose Run Custom Test.
6. If the test fails, fix any errors in the utterance or capability definition.
7. To rerun the utterances, choose Rerun.

Test in the sandbox

You can run ad-hoc testing in the sandbox at any time to validate the generated utterances and test errors and edge cases. Use the results to modify the generated or custom utterances.

To run end-to-end testing in the sandbox

1. To access the sandbox, choose Sandbox from the menu, or expand the Sandbox panel at the bottom of the page.
2. In the UTTERANCE INPUT field, enter one natural language utterance, and then choose Run.
3. In the RESPONSE panel, validate that Alexa sent the expected directive with the expected properties and property values.

Next steps

After your custom capabilities and device pass batch evaluation, you're ready to publish. Continue with Publish a Device with Custom Capabilities to generate the UI for the Alexa app and submit your device for review.

Related topics

• Add Custom Capabilities with the Smart Home AI Toolkit
• Build Custom Capabilities
• Implement Smart Home Add-on Code
• Test and Debug Your Smart Home Add-on

Last updated: Jul 22, 2026