Upgrade Guide

How to upgrade ark-operator between versions — CRD migrations, breaking changes, and safe upgrade procedures.

General upgrade procedure

ark-operator follows a CRDs-first upgrade pattern. Always apply CRDs before upgrading the operator itself.

# 1. Apply new CRDs first
kubectl apply -f https://raw.githubusercontent.com/arkonis-dev/ark-operator/v<new>/config/crd/bases/

# 2. Wait for CRDs to be established
kubectl wait --for=condition=established crd/arkteams.arkonis.dev --timeout=30s

# 3. Upgrade the operator
helm upgrade ark-operator arkonis/ark-operator --version <new>

# 4. Verify the operator came up
kubectl rollout status deployment/ark-operator -n ark-system

Never upgrade the operator before applying the CRDs — the operator will crash-loop if it references a CRD schema that doesn’t exist yet.

v0.8 → v0.9

Breaking changes

None. v0.9 is backwards-compatible with all v0.8 resources.

New in v0.9

  • ark retry <team> — reset failed steps and resume without re-running succeeded steps
  • ark trigger <team> --input <json> — re-run a team from scratch with new input
  • systemPromptRef on role definitions — load prompts from ConfigMap or Secret keys
  • Admission webhook for prompt size (50 KB warning, 512 KB deny per role)
  • agentExtraEnv Helm value — cluster-wide env vars for all agent pods
  • Proactive token budget enforcement in the agent runtime

Migration steps

No CRD migrations required. Existing ArkTeam and ArkAgent resources continue to work unchanged.

If you were injecting API keys via --set apiKeys.openaiApiKey=sk-..., migrate to the existingSecret pattern before upgrading — the --set path still works but is deprecated:

kubectl create secret generic ark-api-keys \
  --from-literal=OPENAI_API_KEY=sk-... \
  --namespace ark-system

helm upgrade ark-operator arkonis/ark-operator \
  --version 0.9.0 \
  --set apiKeys.existingSecret=ark-api-keys

v0.7 → v0.8

Breaking changes

None. v0.8 adds observability (OTel metrics + traces) without changing any CRD schemas.

New in v0.8

  • OTel metrics and distributed tracing (OTLP export)
  • ark trace <task-id> CLI command
  • --trace flag on ark run
  • Structured audit log events (TaskStarted, TaskCompleted, TaskFailed, TaskDelegated)

Migration steps

No CRD migrations required.

To enable OTel export, set OTEL_EXPORTER_OTLP_ENDPOINT in your operator and agent pods:

helm upgrade ark-operator arkonis/ark-operator \
  --set otel.endpoint=http://otel-collector.monitoring.svc.cluster.local:4318

v0.6 → v0.7 (ArkFlow → ArkTeam rename)

Breaking changes

ArkFlow was removed. ArkTeam is the single execution primitive as of v0.9 (unified in RFC-0005). ArkFlow objects will not be recognized by v0.7+ operators.

Migration

Translate each ArkFlow to an ArkTeam with spec.pipeline:

Before (ArkFlow):

apiVersion: arkonis.dev/v1alpha1
kind: ArkFlow
metadata:
  name: research-then-summarize
spec:
  steps:
    - name: research
      arkAgent: research-agent
      inputs:
        topic: "{{ .pipeline.input.topic }}"
    - name: summarize
      arkAgent: summarizer-agent
      dependsOn: [research]
      inputs:
        content: "{{ .steps.research.output }}"
  output: "{{ .steps.summarize.output }}"

After (ArkTeam):

apiVersion: arkonis.dev/v1alpha1
kind: ArkTeam
metadata:
  name: research-then-summarize
spec:
  output: "{{ .steps.summarize.output }}"
  roles:
    - name: research
      model: gpt-4o-mini
      systemPrompt: "You are a research agent."
    - name: summarize
      model: gpt-4o-mini
      systemPrompt: "Summarize the input."
  pipeline:
    - role: research
      inputs:
        topic: "{{ .input.topic }}"
    - role: summarize
      dependsOn: [research]
      inputs:
        content: "{{ .steps.research.output }}"

Key changes:

  • steps[].name → pipeline[].role (references a role defined in spec.roles)
  • steps[].arkAgent → agent config is now inline in the role definition
  • pipeline.input.topic → input.topic (the .pipeline. prefix was removed)
  • Apply ArkTeam CRD first, migrate objects, then delete ArkFlow objects before upgrading the operator

Checking for CRD schema changes

To diff the CRD schemas between two versions:

kubectl get crd arkteams.arkonis.dev -o yaml > current-crd.yaml
curl -s https://raw.githubusercontent.com/arkonis-dev/ark-operator/v<new>/config/crd/bases/arkonis.dev_arkteams.yaml > new-crd.yaml
diff current-crd.yaml new-crd.yaml

Fields added with omitempty are backwards-compatible. Removed required fields or changed types require a migration.

See also