Troubleshooting
FAQ
Symptoms to causes: pages, model, licence, permissions, results, integrations, performance.
Three places to look first: the banner at the top of the app (licence / engine / channel notices), the error code of the call in Audit log, and index=_internal sourcetype=splunkd component=PersistentScript copilot (Python exceptions from the handlers).
| Symptom | Cause / fix |
|---|
| Blank page or the old UI | Splunk Web caches static assets until the app build number changes. Hard-refresh (Ctrl+Shift+R); if the build did not change after an upgrade, splunk restart splunkweb |
| Splunk's own "Something went wrong" bar at the top | The pre-1.6.0 loader started two copies of React for lazily loaded pages. Upgrade to 1.6.0+ |
| Dialogs or drop-downs do nothing | Same loader hid the overlay layer before 1.6.0. Upgrade |
| Language switch has no effect | The switch reloads the page; the AI's output language is response_language |
| Symptom | Cause / fix |
|---|
| "Test connection" fails | The URL needs the version path (Ark: …/api/coding/v3, not …/api/v3); the key is not entitled to the model; port 443 from the search head is blocked |
| Every call takes 30 s or more | Thinking mode is on. AI settings → provider card → thinking mode "Off"; on Ark the same prompt went 29 s → 5.8 s |
| A model returns 400 on every call | It rejects the thinking parameter (kimi / glm / minimax on Ark). 1.8.0+ sends parameters per model family; on older versions set thinking mode to "Model default" |
| Investigation / platform interpretation times out | No cap on tool calls per turn (< 1.8.0). Upgrade, or lower [agentic] max_tool_calls. The requests fallback makes one attempt only |
Engine shows requests fallback | Fallback package, Splunk < 10.2, or missing bin/lib/ dependencies (the reason is on AI settings). Nothing is missing |
| Wrong output language | [llm] response_language: auto follows the browser, zh / en fixed |
llm_error | The model returned something other than JSON. Single-quoted Python literals are tolerated since 1.7.0; if it persists, change model or turn thinking off |
| Symptom | Cause / fix |
|---|
| "Bound to a different machine" right after a successful activation | Before 1.6.0 phone-home fingerprinted with the placeholder GUID in the licence. Upgrade, deactivate, activate again |
machine_id_unreadable | The container has no /etc/machine-id. Write a fixed 32-hex value at start (never a random one per start) |
Detection rules feature_sealed with a valid licence | The licence excludes detection, an offline licence was issued without the key (--with-keyring), or the feature key did not arrive with activation (deactivate and activate once) |
| "In grace period" | No successful heartbeat for 24 hours. Check outbound to license.reallysec.com:443; revert any [license_server] url override in local/ |
Trial request trial_already_requested | One per host (by fingerprint). Ask sales for an extension |
Trial request trial_server_unreachable | The licence server is unreachable or the self-serve endpoint is not yet open |
| Symptom | Cause / fix |
|---|
Every AI request forbidden | The user has no rst role (missing copilot_use). Assign rst_copilot_viewer or above |
| A button says "needs analyst / admin" | Tier too low, see Roles and permissions |
Save as Report returns forbidden (1.8.0) | The whole copilot_deploy endpoint sat behind the admin gate. Use 1.8.0 with the verification fix or later |
Several platform-health checks unknown | No read access to licenser / cluster; under a Free licence the cluster endpoint returns 402 |
| Symptom | Cause / fix |
|---|
| Always empty | The allow-list omits the index; data is outside the time window (the card shows the index's range); the log wording differs from the question (the card lists counts with each condition relaxed) |
| The model invents fields or owners | It does not know this instance's lookups / macros. Runbooks → Sync Splunk objects |
| Different results for the same question | Generation is non-deterministic; copy the SPL or save it as a favourite to fix it |
| Same old answer after a correction | "Ignore" was clicked on the confirmation bar; or a cache hit (1.9.0 folds knowledge-base hits into the cache key) |
| Symptom | Cause / fix |
|---|
| No "Investigate with Copilot" in Incident Review | workflow_actions.conf needs splunk restart; Incident Review only shows actions valid for the event_id field |
Write-back es_not_installed | No ES on this instance, or SplunkEnterpriseSecuritySuite is not visible to the user |
Deploy deploy_rejected: alert.severity | A Splunk Free licence has no alerting. Enterprise works |
| Channel test fails | The row shows the channel's own message: bad signature (Feishu 19001), host not allow-listed, expired URL |
Ticket tickets_not_configured / ticket_auth_failed | Ticketing not configured under Outbound channels, or credentials rejected; on-prem hosts must be in [tickets] allowed_hosts |
| Scheduled jobs (baseline / KB sync / auto-triage) never run | Scripted inputs run as splunk-system-user, absent under a Free licence; on Cloud check the scripted-input policy |
| MCP all 401 / not responding | See MCP → Troubleshooting |
1.9.0 test box (Volcengine Ark ark-code-latest, thinking off, agentic): NL → SPL 5–25 s (0.4 s cached), explain 15 s, detection rule 19 s, triage of 100 alerts 35 s, investigation 51–58 s (9–10 tool calls), platform interpretation 25–32 s, onboarding generation 21–29 s. Anything markedly slower: check thinking mode and [agentic] max_tool_calls first.
