Schedule a Call

How to Use AI Orchestrator

Updated Oct 20, 2026 8 min read

Article verified for Release 16.2 on October, 2026.

The AI Orchestrator is an AI-powered feature that enables users to create and manage users, questions, question pools, tests, surveys, training courses, and testing sessions using natural-language instructions, reducing manual work and accelerating content creation within the application.

The AI Orchestrator allows you to perform actions in the application using simple, natural-language commands. Instead of manually navigating through different pages, you can instruct the Orchestrator to complete tasks for you, such as creating users, tests, or surveys, assigning candidates to groups or sessions, and more.

The Orchestrator helps streamline workflows and speeds up repetitive actions by executing commands directly from a single interface.

Access and Availability: If you cannot access the AI Orchestrator, contact your system administrator to verify that the feature is enabled and that your role has the required permissions. The AI Orchestrator is available as part of the Enterprise Edition (EE) package. If you are unsure whether your environment includes this package, contact sales@youtestme.com or your designated account manager.

Enabling the Orchestrator

AI Orchestrator must first be enabled for your environment by the YouTestMe team. Once enabled, the AI Orchestrator configuration becomes available under System → Integrations → AI Model.

Configure the Integration

The first step is to configure the integration on your application.

  1. Navigate to the System page.
  2. Open the Integrations section.
  3. Select the AI Model tab.
  4. Locate the feature you want to configure (in this case, AI Orchestrator) and click the Edit icon.
  5. In the configuration window:
    • Select the AI provider (OpenAI or Local (Llama)). Note: The use of local Llama models is currently not recommended due to their limited token capacity and insufficient performance.
    • Enter the OpenAI API secret key .
    • Define the OpenAI API URL.
    • Enter the OpenAI API model (for example, gpt-4).
  6. Click the Save button to apply the changes.

Enabling the Permission

  1. Navigate to Users > Roles and Permissions.
  2. Locate and edit the desired role.
  3. Enable the Access AI Orchestrator permission.
  4. Click Update to save your changes.

Important Note

The Access AI Orchestrator permission allows users to access the Orchestrator. However, additional permissions are required for the Orchestrator to create content and allow users to view the content they create.

Security Recommendation – For security reasons, access to the AI Orchestrator should be restricted to the Admin role and should not be granted to other roles without careful consideration. The Orchestrator can perform a wide range of actions on behalf of the user, including creating and managing content. Restricting access helps prevent unauthorized actions and ensures that the feature is used only by trusted users.

Required Permissions – In addition to the Access AI Orchestrator permission, the role must have the following permissions enabled so that the Orchestrator can create content and users can view the content they create:

  • Access the User Profiles page.
  • Create new users.
  • View any user’s personal information.
  • Grant roles to users.
  • Access the User Groups page.
  • Create new subgroups.
  • Create and delete top-level user groups.
  • Access the Question Pools page.
  • Create new question pools.
  • Access Manage Tests, Manage Surveys, Grading Scales, and Training Courses.
  • Create new tests, surveys, and training courses.
  • View questions.
  • Fully manage their own tests, surveys, and training courses.
  • Access the Testing Center dashboards.
  • Create testing sessions.
  • Access the Candidates page.
  • Access the Testing Sessions page.
  • Access the REST API.

The AI Orchestrator can perform only actions allowed by the user’s assigned permissions. Ensure that all required permissions are enabled for the assigned role so that the Orchestrator can perform the necessary actions and users can access the content they create.

Accessing the Orchestrator

To access the Orchestrator, follow these steps:

  1. Click AI Orchestrator in the left-side menu to open the page.
  2. When the page opens, you will see a large text box labeled Your command. Enter your instruction in this box using natural language to describe the action you want the Orchestrator to perform. While the box is empty, it displays examples of commands you can use.
  3. Tips and Tricks – Use this section to explore example commands for each type of operation, along with guidance on how to phrase your instructions.
  4. Click Send command to submit your instruction to the AI agent and execute the requested action.

Using the Orchestrator

1. Enter the command you want to execute in the text box. Describe what you want the Orchestrator to do using clear and specific instructions. For example, “Create a Geography test with 10 questions.”

2. Tips and Tricks – If you are unsure how to write a command, open the relevant section to see examples and instructions for the action you want to perform. You can use these examples to help write your own command.

3. Send Command – Once you have entered your command, click Send command to submit it. The Orchestrator will process your request and display the result below the command box. This usually takes a few seconds, but commands using Local Llama may take several minutes.

4. Review the Result – If the command is successful, the Orchestrator displays a confirmation message with links to the created records. Click a link to open the record in GetCertified. Newly created tests are saved as Draft and must be reviewed and published before candidates can access them.

5. If the Command Fails – The Orchestrator displays a message explaining why the command could not be completed and what you can do to resolve the issue. Read the error message and the What to do section, then correct your command and try again. For example, if a question pool name already exists, enter a different name.

Limitations

Note the following limitations when using the AI Orchestrator:

  • Supported Question Types – The AI Orchestrator supports generating the following question types: Single Choice (SNC), Multiple Choice (MLC), True/False (TFC), Essay (ESY), and Open Question (OPQ).
  • Question creation limit: You can create a maximum of 10 questions per prompt. If you need more questions, submit additional prompts.
  • Duplicate avoidance limitation: The method for avoiding duplicate questions is not guaranteed when working with a large question pool containing more than 100 questions.
  • Prompt clarity required: Commands must be specific and sufficiently detailed. Generic prompts may fail or return incomplete results.
  • Draft-only output: Newly created content (for example, tests) is saved as Draft by default and must be reviewed, configured, and published manually before it becomes available to candidates.

These limitations are designed to ensure accuracy, security, and controlled content creation within the platform.

Examples

Example 1: Successfully Executed Command

Prompt
Create a “TEST NAME” test with 10 questions.

Result
When the command is executed successfully, the AI Orchestrator displays a confirmation message showing that the test was created. The result includes the test name as a clickable link, which you can select to open the test in GetCertified, and the test ID, which identifies the created test.

Below the confirmation message, the result displays the original command and a list of the 10 questions generated for the test. This allows you to see what the AI Orchestrator created and review the questions directly in the results section.

A message at the bottom explains that the test is saved as a Draft. Before candidates can take the test, open it to review its settings and questions, publish it, and create a test session.

Next steps:

The test has been created, but it is not yet available to candidates. To prepare the test for candidates:

  1. Navigate to Tests > Manage tests.
  2. Use the ID from the success response to locate your test (see the above image).
  3. The test is saved as a Draft, meaning it is not visible to any candidate.
  4. Select the Testing sessions tab to create a new session. For more information, refer to this article.
  5. Once you are ready, select the Publish button to make the test available.

Example 2: Unsuccessful Command Execution

Prompt
Create a

Result
The command could not be executed because the instruction is incomplete. The AI Orchestrator displays a This command is not supported message explaining that nothing was created because the AI agent could not determine what action to perform.

The result shows the original command under Your Command and provides a What to Do section with guidance on how to fix the issue. It explains that the agent can create tests, test sessions, questions, question pools, users, and groups. To try again, open Tips and Tricks, select the example closest to your intended action, and replace the example values with your own.

Troubleshooting Failed Commands

If the AI Orchestrator cannot execute a command because of an AI model configuration issue, the result explains the problem and provides guidance on how to fix it.

Depending on the issue, you may see messages such as:

  • That AI model is not available – The configured model name is incorrect or unavailable to your OpenAI account.
  • The AI API key was refused – The API key is invalid or expired.
  • The AI model could not be reached – The AI service could not be reached.
  • No AI model is set up for this page – No AI model is configured for the AI Orchestrator.
  • The local AI model is not installed – The configured Local Llama model is incorrect or is not installed on the server.
  • The local AI model is not running – The Local Llama server cannot be reached.

If you have permission to update the configuration, click Open AI Model settings in the result and correct the settings. Save your changes and submit the command again. You do not need to reload the page.

If you cannot access the AI model settings, ask your administrator to check the configuration. For other error messages, follow the instructions displayed in the result.

Conclusion

The AI Orchestrator streamlines administrative tasks and accelerates the creation of users and content within the application. By using natural-language commands, administrators can create question pools, tests, surveys, training courses, and testing sessions, reducing manual work and simplifying everyday workflows. This helps save time and improves efficiency when managing users and preparing assessment content.

Additional resources:

Previous How to Ensure Email Is Used as the Username
Next How to Take Virtual Interview – Candidate’s Perspective

Was this article helpful?

How to Ensure Email Is Used as the UsernameHow to Take Virtual Interview – Candidate’s Perspective