Upgrade and backup

This page covers app upgrade, backup, migration and downgrade.

Data locations

Upgrade

Online update

As an admin, open Settings; the Version card checks for updates on its own. The app reads a signed update manifest from the GitHub Release; when a newer version exists the button reads Update to X. Clicking it:

1. Downloads the new .spl of the same kind as the installed package (selfcontained or fallback).
2. Checks its size and sha256 against the signed manifest; nothing is installed if the signature or a check fails.
3. Backs up the current version to $SPLUNK_HOME/var/run/rst_splunk_ai_copilot/update/, then installs the new one.
4. Click Restart Splunk now to finish.

To roll back, upload the backup .spl from Manage Apps. The admin needs install_apps / edit_local_apps and restart_splunkd; the built-in admin role has them.

The in-app update is controlled by [update] enabled in copilot.conf (from 2.0.16). The GitHub fallback and selfcontained packages have it on; an admin can turn it off with [update] enabled = 0 in local/copilot.conf. The splunkbase package has it off and sets check_for_updates = 1: new versions show in Splunk's Apps > Manage Apps and are installed from there, and the Version card shows a notice instead of the update button. An admin can turn the in-app update back on with [update] enabled = 1 in local/copilot.conf; later in-app updates then install the GitHub fallback package, which leaves the Splunkbase channel.

Search-head cluster members cannot install apps from a file, and the Version card says so; update a cluster on the deployer. Splunk Cloud is not supported yet; see Install.

On an air-gapped network where Splunk cannot reach GitHub, point release_base_url in local/copilot.conf at an internal mirror:

[update]
# default: https://github.com/Reallysec/RST-AI-Copilot-for-Splunk/releases
release_base_url = https://mirror.example.internal/rst-splunk-ai-copilot/releases

The mirror must keep the GitHub Releases layout: the manifest at <release_base_url>/latest/download/release-manifest.token and the packages under <release_base_url>/download/v<version>/. Only the manifest signature decides what installs, so a mirror cannot change what gets installed.

Manual upgrade

Install the new .spl over the old one and restart. default/ is replaced; local/ and the KV store are untouched. Checks after the upgrade:

1. Check the engine card on AI settings under Settings. The selfcontained package refreshes bin/lib/, replacing anything vendored by hand.
2. Check the licence page. An online licence picks up any new feature keys at its next heartbeat; an offline licence has its feature set fixed in the file, so a new paid feature needs a re-signed .lic.
3. New KV collections are declared in default/collections.conf and created on first use. Field-type corrections in collections.conf (1.6.0 changed timestamps from number to string) apply with the package; old documents stay as they are.
4. Hard-refresh the browser once. Splunk Web caches static assets until the app build number changes.

Upgrading from 2.0.2 or earlier to 2.0.3

2.0.3 keeps data per signed-in user and tightens the factory defaults. After the upgrade:

• Earlier conversations and favourites are not visible. They used to be shared by every user (stored under nobody); from 2.0.3 each user has their own, and the old data is not migrated.
• An installation that never saved its LLM settings in the UI and relied on the factory default address reports that the AI model is not configured yet. Fill in the model endpoint, model name and API key on the AI settings page to restore it.
• On Splunk Enterprise, sign in again after choosing Restart Splunk now.

Upgrading from 2.0.3 to 2.0.4

• The offline grace drops from 14 days to 3, and offline_grace_days in copilot.conf can only shorten it. Licence server replies must carry a signed receipt.
• The MCP index tools need a low-privilege search account configured on the MCP page first; until then they are disabled. See MCP.
• The read-only SPL check is stricter: deploying or previewing detection rules that use map, macros, dbx* or fit is refused. Check existing detection rules for these before upgrading.
• Alert notifications and ServiceNow / Jira tickets are masked per [masking] mode; if masking fails, nothing is sent.

Upgrading from 2.0.4 to 2.0.5

• Knowledge-base text and text sent to the embeddings endpoint are masked per [masking] mode. After the upgrade, existing vectors are re-embedded in the background under the current mode, up to 64 per run (copilot.conf [embeddings] reindex_batch; 0 turns the automatic run off). This calls the embeddings endpoint, and retrieval is slightly weaker until it finishes.
• The selfcontained package supports Linux x86_64 only. Installed on another platform, licence verification fails and paid features are unavailable; use the fallback package on Windows or ARM (aarch64) hosts, or when installing from Splunkbase.

Upgrading from 2.0.6 to 2.0.7

• The app list shows the app as RST AI Copilot for Splunk. The app ID rst_splunk_ai_copilot is unchanged, so settings and data are not affected.
• After the upgrade, a background run rescans the knowledge base once and embeds content that has no vector, up to [embeddings] reindex_batch items per run. This calls the embeddings endpoint.
• Embeddings calls count toward the [usage] monthly budget. If the price list has no entry for the embeddings model, the default price applies.
• New setting copilot.conf [embeddings] max_input_chars (default 8000): longer text is cut before it is sent. Lower it for local models with a small input limit.

Upgrading from 2.0.7 to 2.0.8

• On the first run, existing 2.0.7 knowledge-base vector rows get a vec_sig stamp in one batch KV write. This makes no embeddings calls and re-embeds nothing; a re-index runs automatically only if the embeddings model or endpoint changes later.
• Embeddings usage is no longer priced at the default LLM token price: without an explicit token_prices entry for the embeddings model it shows as unpriced.
• ES 8 custom statuses: ES only accepts a status transition that is enabled for the role. Enable it in reviewstatuses.conf for the roles that need it (for example ess_analyst, ess_admin): transition_reviewstatus-<from>_to_<to> = enabled under [transitioner:<role>]. Otherwise ES rejects the update and the app shows ES's own reason ("... the transition is not allowed").
• The workflow action is now labelled "Investigate with Copilot" by default, with a zh_CN locale pack ("用 Copilot 调查"). The ES 8 Analyst Queue More menu does not localize workflow actions; that is a Splunk limitation.
• Also in this release:
  • Investigations and triage write back to ES 8.
  • ES Incident Review write-back and audit forwarding (syslog / webhook) are out of Beta; LLM failover, ticketing and Slack/Teams stay Beta.
  • Audit events add provider and served_model (the model that answered after failover); usage cost is priced by served_model.
  • An expired or revoked license falls back to Community (EULA 1.2).

Upgrading from 2.0.9 to 2.0.10

No extra steps. Changes in this release:

• The License page accepts the .lic file from the license server as-is, uploaded or pasted. An uploaded file shows its license ID and expiry, not the token; a .lic for another product or in the wrong format gets a specific error.
• The ES alert list shows each notable's current ES status (including custom statuses) and Incident Review urgency, and the detail drawer shows status, owner and disposition. Batch triage ranks ES notables by their current urgency.
• ES write-back offers the statuses configured in ES, including custom ones, and falls back to the five ES defaults when the list cannot be read.
• The AI settings page shows and edits [embeddings] max_input_chars.
• | copilot and | splexplain report errors in English when the lang= argument, the Copilot UI language, the Splunk user language or copilot.conf response_language is English.
• A /etc/machine-id in a non-UTF-8 encoding no longer crashes host fingerprinting.

Upgrading from 2.0.10 to 2.0.11

Single search heads are not affected by the licensing change. Hosts bound by the old fingerprint keep working without re-activation, and the upgrade uses no extra node. On an online license the host moves to the new fingerprint by itself at the next session renewal, with nothing to do; renewal happens when the session has less than 7 days left, so it completes within about 23 days, and every feature keeps working meanwhile.

On a search-head cluster, licensing is now per member. Make sure the license covers every member before you upgrade:

• Online license: before 2.0.11 the whole cluster counted as one node; now each member is a node. Members beyond the license's max_nodes are refused and lock once their offline grace runs out, which can be right after the upgrade. On 2.0.11 the UI reports that the offline grace is used up and the app log records license activation limit reached; from 2.0.12 the License page says the node limit is reached. Check max_nodes against the member count, or move to Enterprise (no node limit), first.
• Offline license: one per member, issued for that member's fingerprint. An offline license imported on 2.0.10 or earlier is valid only on the member it was issued for; import each other member's own license after upgrading.
• After the upgrade each member activates as its own node on its first heartbeat (about 5 minutes), so sealed features can be locked briefly.

For cluster licensing, see Settings > License.

Also in this release:

• Operations reports can be in English. A report you generate follows the UI language; scheduled reports use copilot.conf [reports] language (zh or en, default zh). Reports archived from 2.0.11 on open in the viewer's UI language; older archives are unchanged. Duplicated section numbers in the Chinese report are fixed.
• New host fingerprint recipe, the same as the license server SDK. The MAC counts only when there is no machine-id. A container with no machine-id and no readable MAC gets a random host ID saved in the app directory instead of failing activation. For containers, see Running in Docker.
• An offline-activated license no longer contacts the license service (it used to download the revocation list daily).
• The license check no longer rewrites stored state on almost every request.
• EULA 1.3: company-level terms aligned across the RST product family (liability cap, Chinese version prevails, subscription-term warranty, IP indemnity for paid editions, audit, export control, 30-day data export after termination, privacy and cross-border terms); the data-disclosure section matches what the app sends.
• The product is named RST AI Copilot for Splunk throughout the app, notifications and reports.

Upgrading from 2.0.11 to 2.0.12

• After the upgrade, every user must accept EULA 1.4 the first time they open the app: tick "I have read and agree to the EULA v1.4" and select Accept and continue. Until then the app's REST endpoints refuse that user's requests. This applies to every edition, including Community, and again whenever the EULA version changes. The | copilot and | splexplain search commands, the MCP server and background tasks are not affected.
• For a script or integration account that calls the REST endpoints directly, sign in to Splunk Web with that account once, open the app and accept.
• From 2.0.12, any online host still bound to its pre-2.0.11 fingerprint (activated on 2.0.10 or earlier, including hosts that upgraded to 2.0.11 and have not renewed yet) moves to the current fingerprint about 5 minutes after the upgrade, without using an extra node. Offline licenses are not affected.

Also in this release:

• When the license's node limit is used up (for example an extra search-head cluster member), the License page shows "License node limit reached" as soon as the host is refused, and paid features stop at the end of the grace period. Earlier versions reported this as the license server being unreachable.
• License errors such as a clock rollback show their real reason in the UI language, instead of a generic request failure.
• Scheduled report push works: each report is pushed once per period to the notification targets bound to that period, and failed pushes are retried. Before, nothing was sent. Requires the reports entitlement.
• New setting Scheduled report language under Settings > Reports; scheduled reports and their push cards follow it.
• Switching the UI language re-renders the open report in the new language; a custom time range offers to regenerate.
• English error messages from the search commands use English feature names.
• EULA 1.4: adds in-app click-accept; Section 13 states that, for a paid license, confirming a new version during the subscription term does not change which version applies to that term; more detail on the host fingerprint, including the locally generated host ID; the license service is hosted in the Hong Kong SAR.

Upgrading from 2.0.12 to 2.0.13

• Generic webhook report pushes use the same field names in every language; only the card text is translated. In English mode 2.0.12 used English field names (such as Period and AI calls); 2.0.13 returns to the fixed names. Integrations that parse the English names must switch back.

Also in this release:

• A scheduled report missed while Splunk was down is pushed once after it comes back. Push settings saved before 2.0.13 start catching up only after the first push following the upgrade, so the upgrade never sends a report twice.
• Saving an unknown time zone for scheduled reports is now an error, instead of silently running in UTC (when the Splunk Python runtime has a time-zone database).
• The License page shows "Expiring soon" 7 days before expiry, instead of 30.

Upgrading from 2.0.13 or earlier to 2.0.14

The packages are renamed to RST-AI-Copilot-for-Splunk-<version>-{selfcontained,fallback}.spl, so the in-app online update fails once (the download returns 404). Upgrade by hand:

1. Download 2.0.14 from the Releases page of Reallysec/RST-AI-Copilot-for-Splunk on GitHub, the same kind as the installed package.
2. In Splunk Web open Apps > Manage Apps > Install app from file, choose the file and check "Upgrade app".
3. Restart Splunk after the install.

The app id rst_splunk_ai_copilot is unchanged, so license, data and settings carry over. Later versions update online again.

Also in this release:

• LLM failover is generally available and no longer Beta.
• Ticketing supports Jira Data Center with a personal access token (PAT); the host must be listed in copilot.conf [tickets] allowed_hosts. Jira Cloud tickets keep bold text and links; ServiceNow incidents no longer set non-default category and contact type values. Ticketing stays Beta.
• Trusted time for license checks uses HTTPS only; NTP (UDP) is no longer used.
• License page: features that cannot run on this host show as locked with a hint naming the cause, including on hosts without a readable hardware fingerprint; legacy licenses show "All features".
• Scheduled report push: a target bound after the schedule was enabled no longer triggers a catch-up push; a hint shows when the push hour falls on the next local day; a hand-edited schedule no longer stops pushes.
• The EULA dialog's language toggle switches the whole dialog.
• A failed update download reports the HTTP status or "update source unreachable" instead of an internal error.

Upgrading from 2.0.14 to 2.0.15

No action needed; the online update works. Right after a splunkd restart, while the KV store is still starting, users who already accepted the EULA are no longer asked to accept it again, and saving push or notification settings (targets, SMTP, schedule) no longer overwrites the stored configuration with defaults. Such a save now fails with "KV Store is not available right now" and succeeds on retry.

Upgrading from 2.0.15 to 2.0.16

No action needed. Also in this release:

• New splunkbase package; see Online update.
• Jira Data Center ticket body: wiki markup in alert data (strike-through, underline, superscript, subscript, images, headings, and lines that would close a code block) is shown as plain text instead of being rendered; dates and hostnames no longer show stray backslashes.
• License page: non-admin users see the admin-only notice on first load instead of a backend error. An offline license bound to another host now says so and points to Settings > License.

Backup

Migrating to a new search head

1. Install the same app version on the new machine.
2. Copy local/, etc/passwd and splunk.secret, or re-enter the secrets in the UI on the new machine.
3. Import the KV collections.
4. Deactivate on the old machine and activate on the new one.

Downgrade

Install the older .spl over the current one. Collections and configuration keys added by a newer version are ignored by the older one; fields the older version does not know are ignored as well. Versions before 1.6.0 use a different set of routes, so bookmarks break.