Troubleshooting
FAQ
Causes and fixes by symptom. Pages, model, licence, permissions, results, integrations, performance.
Three places to look first: the banner at the top of the app (licence, engine and channel notices), the error code of the call in Audit log, and index=_internal sourcetype=splunkd component=PersistentScript copilot (Python exceptions from the handlers).
| Symptom | Cause and fix |
|---|
| Blank page or the old UI | Splunk Web caches static assets until the app build number changes. Hard-refresh (Ctrl+Shift+R); if the build did not change after an upgrade, run splunk restart splunkweb |
| Splunk's own Something went wrong bar at the top | The pre-1.6.0 loader started two copies of React for lazily loaded pages. Upgrade to 1.6.0 or later |
| Dialogs or drop-downs do nothing | The same loader hid the overlay layer before 1.6.0. Upgrade |
| Language switch has no effect | The switch reloads the page. The AI's output language is response_language |
| Symptom | Cause and fix |
|---|
| Test connection fails | The URL needs the version path (Ark: …/api/coding/v3, not …/api/v3); the key is not entitled to the model; port 443 from the search head is blocked |
| Every call takes 30 s or more | Thinking mode is on. Set thinking mode to Off on the provider card in AI settings; on Ark the same prompt went from 29 s to 5.8 s |
| A model returns 400 on every call | It rejects the thinking parameter (kimi, glm, minimax on Ark). 1.8.0 and later send parameters per model family; on older versions set thinking mode to Model default |
| Investigation or platform interpretation times out | Before 1.8.0 there was no cap on tool calls per turn. Upgrade, or lower [agentic] max_tool_calls. The requests fallback makes one attempt only |
Engine shows requests fallback | Fallback package, Splunk below 10.2, or missing bin/lib/ dependencies. The reason is on AI settings; nothing is missing |
| Wrong output language | [llm] response_language: auto follows the browser, zh or en fixed |
llm_error | The model returned something other than JSON. Single-quoted Python literals are tolerated since 1.7.0; if it persists, change model or turn thinking off |
| Symptom | Cause and fix |
|---|
| Bound to a different machine right after a successful activation | Before 1.6.0 phone-home fingerprinted with the placeholder GUID in the licence. Upgrade, deactivate, activate again |
machine_id_unreadable | The container has no /etc/machine-id. Write a fixed 32-hex value at start, never a random one per start |
Detection rules feature_sealed with a valid licence | The licence excludes detection, an offline licence was issued without the key, or the feature key did not arrive with activation (deactivate and activate once) |
| In grace period | No successful heartbeat for 24 hours. Check outbound access to license.reallysec.com:443; revert any [license_server] url override in local/ |
| Symptom | Cause and fix |
|---|
Every AI request forbidden | The user has no rst role and lacks copilot_use. Assign rst_copilot_viewer or above |
| A button reports that analyst or admin rights are needed | Tier too low; see Roles and permissions |
Save as Report returns forbidden (1.8.0) | The whole copilot_deploy endpoint sat behind the admin gate. Use 1.8.0 with the verification fix or later |
Several platform-health checks unknown | No read access to licenser or cluster; under a Free licence the cluster endpoint returns 402 |
| Symptom | Cause and fix |
|---|
| Always empty | The allow-list omits the index; data is outside the time window (the card shows the index's range); the log wording differs from the question (the card lists counts with each condition relaxed) |
| The model invents fields or owners | It does not know this instance's lookups and macros. Run Sync Splunk objects in Runbooks |
| Different results for the same question | Generation is non-deterministic. Copy the SPL or save it as a favourite to fix it |
| Same old answer after a correction | Ignore was chosen on the confirmation bar, or a cache hit (1.9.0 folds knowledge-base hits into the cache key) |
| Symptom | Cause and fix |
|---|
| No Investigate with Copilot in Incident Review | workflow_actions.conf needs splunk restart; Incident Review only shows actions valid for the event_id field |
Write-back es_not_installed | No ES on this instance, or SplunkEnterpriseSecuritySuite is not visible to the user |
Deploy deploy_rejected: alert.severity | A Splunk Free licence has no alerting. Enterprise works |
| Channel test fails | The row shows the channel's own message: bad signature (Feishu 19001), host not allow-listed, expired URL |
Ticket tickets_not_configured or ticket_auth_failed | Ticketing not configured under Outbound channels, or credentials rejected; on-prem hosts must be in [tickets] allowed_hosts |
| Scheduled jobs (baseline, KB sync, auto-triage) never run | Scripted inputs run as splunk-system-user, absent under a Free licence |
| MCP all 401 or not responding | See MCP troubleshooting |
1.9.0 test box (Volcengine Ark ark-code-latest, thinking off, agentic):
| Operation | Duration |
|---|
| NL to SPL | 5 to 25 s, 0.4 s cached |
| Explain | 15 s |
| Detection rule | 19 s |
| Triage of 100 alerts | 35 s |
| Investigation | 51 to 58 s, 9 to 10 tool calls |
| Platform interpretation | 25 to 32 s |
| Onboarding generation | 21 to 29 s |
Anything markedly slower: check thinking mode and [agentic] max_tool_calls first.
