This article explains how Smart Probe works at a technical level: what the AI evaluates, what governs its behavior, and what it can and cannot generate. It is intended for researchers who want to understand the system's logic before designing studies.
For setup instructions, see the Smart Probe Text Card authoring guide.
1. What Smart Probe Does and Does Not Do
Smart Probe is designed to probe deeper on what a respondent has already said. It does not originate new lines of inquiry outside the scope the researcher has defined.
What Smart Probe will do
• Ask follow-up questions that build on the respondent's own answer.
• Stay within the topic established by the anchor question.
• Apply conversational goals to shape the direction of follow-ups.
• Enforce avoidance instructions at every probe turn.
• Stop probing once the response meets the thoughtfulness threshold or the probe limit is reached.
• Generate a summary of the full conversational thread.
What Smart Probe will not do
• Introduce new or unrelated topics.
• Exceed the researcher-configured maximum number of probes.
• Ignore avoidance instructions.
• Override display logic or thoughtfulness thresholds.
• Generate probes if the AI is unavailable, the survey continues with the initial open-ended question only.
2. Inputs That Shape Probe Behaviour
The follow-up generation engine draws on five inputs simultaneously at runtime.
Input | How it shapes probe behaviour |
Anchor question | The core topic question set by the researcher. It defines the topical scope for the entire probing thread. |
Respondent's answer(s) | The primary driver of what the follow-up asks. The AI identifies what was said and what could be explored further within scope. |
Conversational goals | Researcher written instructions describing what the probe should aim to uncover. |
Avoidance instructions | Hard constraints specifying topics or question styles to exclude. Applied at every probe turn. |
Full conversation history | The complete thread up to the current point. Used to prevent redundancy, the AI will not re-ask what has already been covered. |
3. Topic Containment
Smart Probe enforces topic containment through two mechanisms.
Anchor-based scoping
The anchor question sets the topical frame for the entire conversation. The AI uses it as the reference point for what is and is not on-topic. Smart Probe cannot introduce new or unrelated topics.
Avoidance instructions
Avoidance instructions specify topics or question styles the AI must not use. These constraints apply at every probe in the thread, not just the first.
Example • Anchor question: 'How has your experience with the product changed over the past six months?' • Avoidance instruction: 'Do not ask about pricing or competitors.' • A respondent mentions a competitor unprompted. • Result: Smart Probe will not follow up on the competitor reference. It probes only within the experience-change topic. |
What avoidance instructions do not do Avoidance instructions govern what Smart Probe asks — they do not prevent respondents from writing whatever they choose in their answers. If a respondent volunteers information outside the configured scope, that content will appear in the response data. The instructions shape probe behavior only. |
4. Thoughtfulness Scoring
The thoughtfulness score determines whether a follow-up question is generated at all. It evaluates the quality and completeness of the respondent's answer.
Scoring dimensions
Each response is scored across 10 dimensions, each on a 0–10 scale:
Dimension | What it evaluates |
Depth of Insight | Does the response go beyond surface-level statements? |
Relevance | Does it stay on-topic with the anchor question? |
Specificity | Does it include concrete details or examples? |
Clarity | Is it logically structured and easy to follow? |
Originality | Does it reflect an authentic, personal viewpoint? |
Critical Thinking | Does it show reasoning or weigh trade-offs? |
Emotional Engagement | Is there genuine feeling or personal context present? |
Breadth | Does it cover more than one angle or perspective? |
Supporting Evidence | Is there justification or rationale provided? |
Constructiveness | Does it offer suggestions or forward-looking ideas? |
Score calculation
The 10 dimension scores are averaged into a final aggregated score. Two rules apply on top of the average:
• If the relevance dimension scores zero, the final score is set to zero regardless of all other scores.
• Short responses that are genuinely high quality receive an upward adjustment, so concise but substantive answers are not unnecessarily probed.
Scores are internal only Thoughtfulness scores are stored as metadata and never shown to respondents. They are available to researchers in exports and reports. |
5. Probe Generation Logic
Redundancy prevention
Before generating each follow-up, the engine reads the full conversation history. It will not re-ask a question already covered in the thread.
Threshold re-evaluation
After each follow-up response, the scoring engine re-evaluates the new answer against the threshold. Probing stops as soon as the threshold is met, even if the maximum probe count has not been reached.
Hard limits on AI autonomy
The following constraints are enforced at a system level and cannot be overridden by model judgment:
• Cannot exceed the researcher-configured maximum probe count.
• Cannot introduce topics not present in the anchor question or respondent answers.
• Cannot ignore or partially apply avoidance instructions.
• Cannot modify display logic or thoughtfulness thresholds at runtime.
6. Versioning and Auditability
Every Smart Probe operation logs the AI model version and prompt version used — for question generation, thoughtfulness scoring, and summarization — at the respondent level.
Version behavior
• All respondents in a study are evaluated by the same model and prompt version.
• If a scoring framework is updated after a study launches, the existing study continues on its original version. The new version applies only to new studies.
• This document is updated when a new scoring version becomes available.
Current model configuration
AI Function | GPT Model | Prompt Version | Key Purpose |
Generate Question | GPT-4.1 Mini | 1.0.0 | Generates context-aware follow-up questions based on respondent answers and researcher-defined goals. |
Thoughtfulness Scoring | GPT-4.1 Mini | 2.0.0 | Evaluates response depth and completeness using the 10-dimension rubric. |
Summarize Messages | GPT-4o | 1.0.0 | Produces a concise synthesis of the full probing thread for reporting. |
7. Language Support
• Smart Probe can be used in any language, but it has been primarily trained and tested in English. Performance is most reliable in English-language studies.
• In other languages, thoughtfulness scoring, follow-up question quality, and summary accuracy may be less consistent.
• For non-English deployments, test the full probe flow in a test chat before deploying to respondents.