refactor: migrate ModalHeader to Content layout

Modal.Header now uses opal Content component for icon + title rendering. Description passed to Content directly with a hidden DialogPrimitive.Description for accessibility. Close button absolutely positioned per Figma mocks.
refactor(opal): split ContentLg into ContentXl + ContentLg
2026-03-10 02:02:41 +00:00 · 2026-03-01 12:06:18 -08:00 · 2026-03-01 12:05:56 -08:00
942 changed files with 12139 additions and 41182 deletions
--- a/.cursor/skills/onyx-cli/SKILL.md
+++ b/.cursor/skills/onyx-cli/SKILL.md
@@ -1,186 +0,0 @@
---
-name: onyx-cli
-description: Query the Onyx knowledge base using the onyx-cli command. Use when the user wants to search company documents, ask questions about internal knowledge, query connected data sources, or look up information stored in Onyx.
---
-
-# Onyx CLI — Agent Tool
-
-Onyx is an enterprise search and Gen-AI platform that connects to company documents, apps, and people. The `onyx-cli` CLI provides non-interactive commands to query the Onyx knowledge base and list available agents.
-
-## Prerequisites
-
-### 1. Check if installed
-
-```bash
-which onyx-cli
-```
-
-### 2. Install (if needed)
-
-**Primary — pip:**
-
-```bash
-pip install onyx-cli
-```
-
-**From source (Go):**
-
-```bash
-cd cli && go build -o onyx-cli . && sudo mv onyx-cli /usr/local/bin/
-```
-
-### 3. Check if configured
-
-```bash
-onyx-cli validate-config
-```
-
-This checks the config file exists, API key is present, and tests the server connection via `/api/me`. Exit code 0 on success, non-zero with a descriptive error on failure.
-
-If unconfigured, you have two options:
-
-**Option A — Interactive setup (requires user input):**
-
-```bash
-onyx-cli configure
-```
-
-This prompts for the Onyx server URL and API key, tests the connection, and saves config.
-
-**Option B — Environment variables (non-interactive, preferred for agents):**
-
-```bash
-export ONYX_SERVER_URL="https://your-onyx-server.com"  # default: https://cloud.onyx.app
-export ONYX_API_KEY="your-api-key"
-```
-
-Environment variables override the config file. If these are set, no config file is needed.
-
-| Variable | Required | Description |
-|----------|----------|-------------|
-| `ONYX_SERVER_URL` | No | Onyx server base URL (default: `https://cloud.onyx.app`) |
-| `ONYX_API_KEY` | Yes | API key for authentication |
-| `ONYX_PERSONA_ID` | No | Default agent/persona ID |
-
-If neither the config file nor environment variables are set, tell the user that `onyx-cli` needs to be configured and ask them to either:
- Run `onyx-cli configure` interactively, or
- Set `ONYX_SERVER_URL` and `ONYX_API_KEY` environment variables
-
-## Commands
-
-### Validate configuration
-
-```bash
-onyx-cli validate-config
-```
-
-Checks config file exists, API key is present, and tests the server connection. Use this before `ask` or `agents` to confirm the CLI is properly set up.
-
-### List available agents
-
-```bash
-onyx-cli agents
-```
-
-Prints a table of agent IDs, names, and descriptions. Use `--json` for structured output:
-
-```bash
-onyx-cli agents --json
-```
-
-Use agent IDs with `ask --agent-id` to query a specific agent.
-
-### Basic query (plain text output)
-
-```bash
-onyx-cli ask "What is our company's PTO policy?"
-```
-
-Streams the answer as plain text to stdout. Exit code 0 on success, non-zero on error.
-
-### JSON output (structured events)
-
-```bash
-onyx-cli ask --json "What authentication methods do we support?"
-```
-
-Outputs JSON-encoded parsed stream events (one object per line). Key event objects include message deltas, stop, errors, search-start, and citation payloads.
-
-Each line is a JSON object with this envelope:
-
-```json
-{"type": "<event_type>", "event": { ... }}
-```
-
-| Event Type | Description |
-|------------|-------------|
-| `message_delta` | Content token — concatenate all `content` fields for the full answer |
-| `stop` | Stream complete |
-| `error` | Error with `error` message field |
-| `search_tool_start` | Onyx started searching documents |
-| `citation_info` | Source citation — see shape below |
-
-`citation_info` event shape:
-
-```json
-{
-  "type": "citation_info",
-  "event": {
-    "citation_number": 1,
-    "document_id": "abc123def456",
-    "placement": {"turn_index": 0, "tab_index": 0, "sub_turn_index": null}
-  }
-}
-```
-
-`placement` is metadata about where in the conversation the citation appeared and can be ignored for most use cases.
-
-### Specify an agent
-
-```bash
-onyx-cli ask --agent-id 5 "Summarize our Q4 roadmap"
-```
-
-Uses a specific Onyx agent/persona instead of the default.
-
-### All flags
-
-| Flag | Type | Description |
-|------|------|-------------|
-| `--agent-id` | int | Agent ID to use (overrides default) |
-| `--json` | bool | Output raw NDJSON events instead of plain text |
-
-## Statelessness
-
-Each `onyx-cli ask` call creates an independent chat session. There is no built-in way to chain context across multiple `ask` invocations — every call starts fresh. If you need multi-turn conversation with memory, use the interactive TUI (`onyx-cli` or `onyx-cli chat`) instead.
-
-## When to Use
-
-Use `onyx-cli ask` when:
-
- The user asks about company-specific information (policies, docs, processes)
- You need to search internal knowledge bases or connected data sources
- The user references Onyx, asks you to "search Onyx", or wants to query their documents
- You need context from company wikis, Confluence, Google Drive, Slack, or other connected sources
-
-Do NOT use when:
-
- The question is about general programming knowledge (use your own knowledge)
- The user is asking about code in the current repository (use grep/read tools)
- The user hasn't mentioned Onyx and the question doesn't require internal company data
-
-## Examples
-
-```bash
-# Simple question
-onyx-cli ask "What are the steps to deploy to production?"
-
-# Get structured output for parsing
-onyx-cli ask --json "List all active API integrations"
-
-# Use a specialized agent
-onyx-cli ask --agent-id 3 "What were the action items from last week's standup?"
-
-# Pipe the answer into another command
-onyx-cli ask "What is the database schema for users?" | head -20
-```
--- a/.github/actions/build-integration-image/action.yml
+++ b/.github/actions/build-integration-image/action.yml
@@ -54,7 +54,6 @@ runs:
      shell: bash
      env:
        RUNS_ON_ECR_CACHE: ${{ inputs.runs-on-ecr-cache }}
-        INTEGRATION_REPOSITORY: ${{ inputs.runs-on-ecr-cache }}
        TAG: nightly-llm-it-${{ inputs.run-id }}
        CACHE_SUFFIX: ${{ steps.format-branch.outputs.cache-suffix }}
        HEAD_SHA: ${{ inputs.github-sha }}
--- a/.github/workflows/deployment.yml
+++ b/.github/workflows/deployment.yml
@@ -151,7 +151,7 @@ jobs:
          fetch-depth: 0

      - name: Setup uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
        with:
          version: "0.9.9"
          # NOTE: This isn't caching much and zizmor suggests this could be poisoned, so disable.
@@ -182,52 +182,8 @@ jobs:
          title: "🚨 Version Tag Check Failed"
          ref-name: ${{ github.ref_name }}

-  # Create GitHub release first, before desktop builds start.
-  # This ensures all desktop matrix jobs upload to the same release instead of
-  # racing to create duplicate releases.
-  create-release:
-    needs: determine-builds
-    if: needs.determine-builds.outputs.build-desktop == 'true'
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    permissions:
-      contents: write
-    outputs:
-      release-id: ${{ steps.create-release.outputs.id }}
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Determine release tag
-        id: release-tag
-        env:
-          IS_TEST_RUN: ${{ needs.determine-builds.outputs.is-test-run }}
-          SHORT_SHA: ${{ needs.determine-builds.outputs.short-sha }}
-        run: |
-          if [ "${IS_TEST_RUN}" == "true" ]; then
-            echo "tag=v0.0.0-dev+${SHORT_SHA}" >> "$GITHUB_OUTPUT"
-          else
-            echo "tag=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Create GitHub Release
-        id: create-release
-        uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # ratchet:softprops/action-gh-release@v2
-        with:
-          tag_name: ${{ steps.release-tag.outputs.tag }}
-          name: ${{ steps.release-tag.outputs.tag }}
-          body: "See the assets to download this version and install."
-          draft: true
-          prerelease: false
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
  build-desktop:
-    needs:
-      - determine-builds
-      - create-release
+    needs: determine-builds
    if: needs.determine-builds.outputs.build-desktop == 'true'
    permissions:
      id-token: write
@@ -252,12 +208,12 @@ jobs:
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6.0.2
        with:
-          # NOTE: persist-credentials is needed for tauri-action to upload assets to GitHub releases.
+          # NOTE: persist-credentials is needed for tauri-action to create GitHub releases.
          persist-credentials: true # zizmor: ignore[artipacked]

      - name: Configure AWS credentials
        if: startsWith(matrix.platform, 'macos-')
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -397,9 +353,11 @@ jobs:
          APPLE_SIGNING_IDENTITY: ${{ env.CERT_ID }}
          APPLE_TEAM_ID: ${{ env.APPLE_TEAM_ID }}
        with:
-          # Use the release created by the create-release job to avoid race conditions
-          # when multiple matrix jobs try to create/update the same release simultaneously
-          releaseId: ${{ needs.create-release.outputs.release-id }}
+          tagName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseBody: "See the assets to download this version and install."
+          releaseDraft: true
+          prerelease: false
          assetNamePattern: "[name]_[arch][ext]"
          args: ${{ matrix.args }}

@@ -426,7 +384,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -468,9 +426,8 @@ jobs:
            ONYX_VERSION=${{ github.ref_name }}
            NODE_OPTIONS=--max-old-space-size=8192
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64,mode=max
@@ -500,7 +457,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -542,9 +499,8 @@ jobs:
            ONYX_VERSION=${{ github.ref_name }}
            NODE_OPTIONS=--max-old-space-size=8192
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64,mode=max
@@ -569,7 +525,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -639,7 +595,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -690,8 +646,8 @@ jobs:
            NEXT_PUBLIC_INCLUDE_ERROR_POPUP_SUPPORT_LINK=true
            NODE_OPTIONS=--max-old-space-size=8192
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64,mode=max
@@ -721,7 +677,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -772,8 +728,8 @@ jobs:
            NEXT_PUBLIC_INCLUDE_ERROR_POPUP_SUPPORT_LINK=true
            NODE_OPTIONS=--max-old-space-size=8192
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64,mode=max
@@ -798,7 +754,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -865,7 +821,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -906,9 +862,8 @@ jobs:
          build-args: |
            ONYX_VERSION=${{ github.ref_name }}
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64,mode=max
@@ -938,7 +893,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -979,9 +934,8 @@ jobs:
          build-args: |
            ONYX_VERSION=${{ github.ref_name }}
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64,mode=max
@@ -1006,7 +960,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1076,7 +1030,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1118,8 +1072,8 @@ jobs:
            ONYX_VERSION=${{ github.ref_name }}
            ENABLE_CRAFT=true
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64,mode=max
@@ -1149,7 +1103,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1191,8 +1145,8 @@ jobs:
            ONYX_VERSION=${{ github.ref_name }}
            ENABLE_CRAFT=true
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64,mode=max
@@ -1218,7 +1172,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1288,7 +1242,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1333,9 +1287,8 @@ jobs:
          build-args: |
            ONYX_VERSION=${{ github.ref_name }}
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64,mode=max
@@ -1368,7 +1321,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1413,9 +1366,8 @@ jobs:
          build-args: |
            ONYX_VERSION=${{ github.ref_name }}
          cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
            type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64
          cache-to: |
            type=inline
            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64,mode=max
@@ -1442,7 +1394,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1507,7 +1459,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1562,7 +1514,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1622,7 +1574,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -1679,7 +1631,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
--- a/.github/workflows/nightly-llm-provider-chat.yml
+++ b/.github/workflows/nightly-llm-provider-chat.yml
@@ -15,11 +15,6 @@ permissions:
 jobs:
  provider-chat-test:
    uses: ./.github/workflows/reusable-nightly-llm-provider-chat.yml
-    secrets:
-      AWS_OIDC_ROLE_ARN: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-    permissions:
-      contents: read
-      id-token: write
    with:
      openai_models: ${{ vars.NIGHTLY_LLM_OPENAI_MODELS }}
      anthropic_models: ${{ vars.NIGHTLY_LLM_ANTHROPIC_MODELS }}
@@ -30,6 +25,16 @@ jobs:
      ollama_models: ${{ vars.NIGHTLY_LLM_OLLAMA_MODELS }}
      openrouter_models: ${{ vars.NIGHTLY_LLM_OPENROUTER_MODELS }}
      strict: true
+    secrets:
+      openai_api_key: ${{ secrets.OPENAI_API_KEY }}
+      anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+      bedrock_api_key: ${{ secrets.BEDROCK_API_KEY }}
+      vertex_ai_custom_config_json: ${{ secrets.NIGHTLY_LLM_VERTEX_AI_CUSTOM_CONFIG_JSON }}
+      azure_api_key: ${{ secrets.AZURE_API_KEY }}
+      ollama_api_key: ${{ secrets.OLLAMA_API_KEY }}
+      openrouter_api_key: ${{ secrets.OPENROUTER_API_KEY }}
+      DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+      DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }}

  notify-slack-on-failure:
    needs: [provider-chat-test]
--- a/.github/workflows/post-merge-beta-cherry-pick.yml
+++ b/.github/workflows/post-merge-beta-cherry-pick.yml
@@ -6,13 +6,11 @@ on:
      - main

 permissions:
-  contents: read
+  contents: write
+  pull-requests: write

 jobs:
  cherry-pick-to-latest-release:
-    permissions:
-      contents: write
-      pull-requests: write
    outputs:
      should_cherrypick: ${{ steps.gate.outputs.should_cherrypick }}
      pr_number: ${{ steps.gate.outputs.pr_number }}
@@ -70,7 +68,7 @@ jobs:

      - name: Install the latest version of uv
        if: steps.gate.outputs.should_cherrypick == 'true'
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
        with:
          enable-cache: false
          version: "0.9.9"
--- a/.github/workflows/pr-desktop-build.yml
+++ b/.github/workflows/pr-desktop-build.yml
@@ -57,7 +57,7 @@ jobs:
          cache-dependency-path: ./desktop/package-lock.json

      - name: Setup Rust
-        uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9
+        uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7
        with:
          toolchain: stable
          targets: ${{ matrix.target }}
--- a/.github/workflows/pr-external-dependency-unit-tests.yml
+++ b/.github/workflows/pr-external-dependency-unit-tests.yml
@@ -160,7 +160,7 @@ jobs:
          cd deployment/docker_compose

          # Get list of running containers
-          containers=$(docker compose -f docker-compose.yml -f docker-compose.dev.yml ps -q)
+          containers=$(docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.opensearch.yml ps -q)

          # Collect logs from each container
          for container in $containers; do
--- a/.github/workflows/pr-golang-tests.yml
+++ b/.github/workflows/pr-golang-tests.yml
@@ -1,56 +0,0 @@
-name: Golang Tests
-concurrency:
-  group: Golang-Tests-${{ github.workflow }}-${{ github.head_ref || github.event.workflow_run.head_branch || github.run_id }}
-  cancel-in-progress: true
-
-on:
-  merge_group:
-  pull_request:
-    branches:
-      - main
-      - "release/**"
-  push:
-    tags:
-      - "v*.*.*"
-
-permissions: {}
-
-env:
-  GO_VERSION: "1.26"
-
-jobs:
-  detect-modules:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      modules: ${{ steps.set-modules.outputs.modules }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
-        with:
-          persist-credentials: false
-      - id: set-modules
-        run: echo "modules=$(find . -name 'go.mod' -exec dirname {} \; | jq -Rc '[.,inputs]')" >> "$GITHUB_OUTPUT"
-
-  golang:
-    needs: detect-modules
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        modules: ${{ fromJSON(needs.detect-modules.outputs.modules) }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # zizmor: ignore[cache-poisoning]
-        with:
-          go-version: ${{ env.GO_VERSION }}
-          cache-dependency-path: "**/go.sum"
-
-      - run: go mod tidy
-        working-directory: ${{ matrix.modules }}
-      - run: git diff --exit-code go.mod go.sum
-        working-directory: ${{ matrix.modules }}
-
-      - run: go test ./...
-        working-directory: ${{ matrix.modules }}
--- a/.github/workflows/pr-helm-chart-testing.yml
+++ b/.github/workflows/pr-helm-chart-testing.yml
@@ -71,7 +71,7 @@ jobs:

      - name: Create kind cluster
        if: steps.list-changed.outputs.changed == 'true'
-        uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # ratchet:helm/kind-action@v1.14.0
+        uses: helm/kind-action@92086f6be054225fa813e0a4b13787fc9088faab # ratchet:helm/kind-action@v1.13.0

      - name: Pre-install cluster status check
        if: steps.list-changed.outputs.changed == 'true'
--- a/.github/workflows/pr-integration-tests.yml
+++ b/.github/workflows/pr-integration-tests.yml
@@ -335,6 +335,7 @@ jobs:
          # TODO(Nik): https://linear.app/onyx-app/issue/ENG-1/update-test-infra-to-use-test-license
          LICENSE_ENFORCEMENT_ENABLED=false
          CHECK_TTL_MANAGEMENT_TASK_FREQUENCY_IN_HOURS=0.001
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=false
          EOF
          fi

@@ -470,13 +471,13 @@ jobs:
          path: ${{ github.workspace }}/docker-compose.log
      # ------------------------------------------------------------

-  onyx-lite-tests:
+  no-vectordb-tests:
    needs: [build-backend-image, build-integration-image]
    runs-on:
      [
        runs-on,
        runner=4cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-onyx-lite-tests",
+        "run-id=${{ github.run_id }}-no-vectordb-tests",
        "extras=ecr-cache",
      ]
    timeout-minutes: 45
@@ -494,12 +495,13 @@ jobs:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

-      - name: Create .env file for Onyx Lite Docker Compose
+      - name: Create .env file for no-vectordb Docker Compose
        env:
          ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
          RUN_ID: ${{ github.run_id }}
        run: |
          cat <<EOF > deployment/docker_compose/.env
+          COMPOSE_PROFILES=s3-filestore
          ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
          LICENSE_ENFORCEMENT_ENABLED=false
          AUTH_TYPE=basic
@@ -507,23 +509,28 @@ jobs:
          POSTGRES_USE_NULL_POOL=true
          REQUIRE_EMAIL_VERIFICATION=false
          DISABLE_TELEMETRY=true
+          DISABLE_VECTOR_DB=true
          ONYX_BACKEND_IMAGE=${ECR_CACHE}:integration-test-backend-test-${RUN_ID}
          INTEGRATION_TESTS_MODE=true
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=true
          EOF

-      # Start only the services needed for Onyx Lite (Postgres + API server)
-      - name: Start Docker containers (onyx-lite)
+      # Start only the services needed for no-vectordb mode (no Vespa, no model servers)
+      - name: Start Docker containers (no-vectordb)
        run: |
          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up \
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml up \
            relational_db \
+            cache \
+            minio \
            api_server \
+            background \
            -d
-        id: start_docker_onyx_lite
+        id: start_docker_no_vectordb

      - name: Wait for services to be ready
        run: |
-          echo "Starting wait-for-service script (onyx-lite)..."
+          echo "Starting wait-for-service script (no-vectordb)..."
          start_time=$(date +%s)
          timeout=300
          while true; do
@@ -545,14 +552,14 @@ jobs:
            sleep 5
          done

-      - name: Run Onyx Lite Integration Tests
+      - name: Run No-VectorDB Integration Tests
        uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # ratchet:nick-fields/retry@v3
        with:
          timeout_minutes: 20
          max_attempts: 3
          retry_wait_seconds: 10
          command: |
-            echo "Running onyx-lite integration tests..."
+            echo "Running no-vectordb integration tests..."
            docker run --rm --network onyx_default \
              --name test-runner \
              -e POSTGRES_HOST=relational_db \
@@ -563,38 +570,39 @@ jobs:
              -e DB_READONLY_PASSWORD=password \
              -e POSTGRES_POOL_PRE_PING=true \
              -e POSTGRES_USE_NULL_POOL=true \
+              -e REDIS_HOST=cache \
              -e API_SERVER_HOST=api_server \
              -e OPENAI_API_KEY=${OPENAI_API_KEY} \
              -e TEST_WEB_HOSTNAME=test-runner \
              ${{ env.RUNS_ON_ECR_CACHE }}:integration-test-${{ github.run_id }} \
              /app/tests/integration/tests/no_vectordb

-      - name: Dump API server logs (onyx-lite)
+      - name: Dump API server logs (no-vectordb)
        if: always()
        run: |
          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_onyx_lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_no_vectordb.log || true

-      - name: Dump all-container logs (onyx-lite)
+      - name: Dump all-container logs (no-vectordb)
        if: always()
        run: |
          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color > $GITHUB_WORKSPACE/docker-compose-onyx-lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color > $GITHUB_WORKSPACE/docker-compose-no-vectordb.log || true

-      - name: Upload logs (onyx-lite)
+      - name: Upload logs (no-vectordb)
        if: always()
        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
        with:
-          name: docker-all-logs-onyx-lite
-          path: ${{ github.workspace }}/docker-compose-onyx-lite.log
+          name: docker-all-logs-no-vectordb
+          path: ${{ github.workspace }}/docker-compose-no-vectordb.log

-      - name: Stop Docker containers (onyx-lite)
+      - name: Stop Docker containers (no-vectordb)
        if: always()
        run: |
          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml down -v
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml down -v

  multitenant-tests:
    needs:
@@ -736,7 +744,7 @@ jobs:
    # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
    runs-on: ubuntu-slim
    timeout-minutes: 45
-    needs: [integration-tests, onyx-lite-tests, multitenant-tests]
+    needs: [integration-tests, no-vectordb-tests, multitenant-tests]
    if: ${{ always() }}
    steps:
      - name: Check job status
--- a/.github/workflows/pr-jest-tests.yml
+++ b/.github/workflows/pr-jest-tests.yml
@@ -31,7 +31,7 @@ jobs:
        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
        with:
          node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning] test-only workflow; no deploy artifacts
+          cache: "npm"
          cache-dependency-path: ./web/package-lock.json

      - name: Install node dependencies
--- a/.github/workflows/pr-playwright-tests.yml
+++ b/.github/workflows/pr-playwright-tests.yml
@@ -268,11 +268,10 @@ jobs:
          persist-credentials: false

      - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
        with:
          node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
+          cache: "npm"
          cache-dependency-path: ./web/package-lock.json

      - name: Install node dependencies
@@ -280,7 +279,6 @@ jobs:
        run: npm ci

      - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
        uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
        with:
          path: ~/.cache/ms-playwright
@@ -461,14 +459,14 @@ jobs:
      # --- Visual Regression Diff ---
      - name: Configure AWS credentials
        if: always()
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2

      - name: Install the latest version of uv
        if: always()
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
        with:
          enable-cache: false
          version: "0.9.9"
@@ -592,108 +590,6 @@ jobs:
          name: docker-logs-${{ matrix.project }}-${{ github.run_id }}
          path: ${{ github.workspace }}/docker-compose.log

-  playwright-tests-lite:
-    needs: [build-web-image, build-backend-image]
-    name: Playwright Tests (lite)
-    runs-on:
-      - runs-on
-      - runner=4cpu-linux-arm64
-      - "run-id=${{ github.run_id }}-playwright-tests-lite"
-      - "extras=ecr-cache"
-    timeout-minutes: 30
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
-        with:
-          node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
-          cache-dependency-path: ./web/package-lock.json
-
-      - name: Install node dependencies
-        working-directory: ./web
-        run: npm ci
-
-      - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
-        with:
-          path: ~/.cache/ms-playwright
-          key: ${{ runner.os }}-playwright-npm-${{ hashFiles('web/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-playwright-npm-
-
-      - name: Install playwright browsers
-        working-directory: ./web
-        run: npx playwright install --with-deps
-
-      - name: Create .env file for Docker Compose
-        env:
-          OPENAI_API_KEY_VALUE: ${{ env.OPENAI_API_KEY }}
-          ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
-          RUN_ID: ${{ github.run_id }}
-        run: |
-          cat <<EOF > deployment/docker_compose/.env
-          ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
-          LICENSE_ENFORCEMENT_ENABLED=false
-          AUTH_TYPE=basic
-          INTEGRATION_TESTS_MODE=true
-          GEN_AI_API_KEY=${OPENAI_API_KEY_VALUE}
-          MOCK_LLM_RESPONSE=true
-          REQUIRE_EMAIL_VERIFICATION=false
-          DISABLE_TELEMETRY=true
-          ONYX_BACKEND_IMAGE=${ECR_CACHE}:playwright-test-backend-${RUN_ID}
-          ONYX_WEB_SERVER_IMAGE=${ECR_CACHE}:playwright-test-web-${RUN_ID}
-          EOF
-
-      # needed for pulling external images otherwise, we hit the "Unauthenticated users" limit
-      # https://docs.docker.com/docker-hub/usage/
-      - name: Login to Docker Hub
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_TOKEN }}
-
-      - name: Start Docker containers (lite)
-        run: |
-          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up -d
-        id: start_docker
-
-      - name: Run Playwright tests (lite)
-        working-directory: ./web
-        run: npx playwright test --project lite
-
-      - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        if: always()
-        with:
-          name: playwright-test-results-lite-${{ github.run_id }}
-          path: ./web/output/playwright/
-          retention-days: 30
-
-      - name: Save Docker logs
-        if: success() || failure()
-        env:
-          WORKSPACE: ${{ github.workspace }}
-        run: |
-          cd deployment/docker_compose
-          docker compose logs > docker-compose.log
-          mv docker-compose.log ${WORKSPACE}/docker-compose.log
-
-      - name: Upload logs
-        if: success() || failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        with:
-          name: docker-logs-lite-${{ github.run_id }}
-          path: ${{ github.workspace }}/docker-compose.log
-
  # Post a single combined visual regression comment after all matrix jobs finish
  visual-regression-comment:
    needs: [playwright-tests]
@@ -707,7 +603,7 @@ jobs:
      pull-requests: write
    steps:
      - name: Download visual diff summaries
-        uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131
        with:
          pattern: screenshot-diff-summary-*
          path: summaries/
@@ -790,7 +686,7 @@ jobs:
    # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
    runs-on: ubuntu-slim
    timeout-minutes: 45
-    needs: [playwright-tests, playwright-tests-lite]
+    needs: [playwright-tests]
    if: ${{ always() }}
    steps:
      - name: Check job status
--- a/.github/workflows/pr-python-checks.yml
+++ b/.github/workflows/pr-python-checks.yml
@@ -8,7 +8,7 @@ on:
  pull_request:
    branches:
      - main
-      - "release/**"
+      - 'release/**'
  push:
    tags:
      - "v*.*.*"
@@ -21,13 +21,7 @@ jobs:
    # See https://runs-on.com/runners/linux/
    # Note: Mypy seems quite optimized for x64 compared to arm64.
    # Similarly, mypy is single-threaded and incremental, so 2cpu is sufficient.
-    runs-on:
-      [
-        runs-on,
-        runner=2cpu-linux-x64,
-        "run-id=${{ github.run_id }}-mypy-check",
-        "extras=s3-cache",
-      ]
+    runs-on: [runs-on, runner=2cpu-linux-x64, "run-id=${{ github.run_id }}-mypy-check", "extras=s3-cache"]
    timeout-minutes: 45

    steps:
@@ -58,14 +52,21 @@ jobs:
        if: ${{ vars.DISABLE_MYPY_CACHE != 'true' }}
        uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
        with:
-          path: .mypy_cache
-          key: mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-${{ hashFiles('**/*.py', '**/*.pyi', 'pyproject.toml') }}
+          path: backend/.mypy_cache
+          key: mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-${{ hashFiles('**/*.py', '**/*.pyi', 'backend/pyproject.toml') }}
          restore-keys: |
            mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-
            mypy-${{ runner.os }}-

      - name: Run MyPy
+        working-directory: ./backend
        env:
          MYPY_FORCE_COLOR: 1
          TERM: xterm-256color
        run: mypy .
+
+      - name: Run MyPy (tools/)
+        env:
+          MYPY_FORCE_COLOR: 1
+          TERM: xterm-256color
+        run: mypy tools/
--- a/.github/workflows/pr-quality-checks.yml
+++ b/.github/workflows/pr-quality-checks.yml
@@ -28,7 +28,7 @@ jobs:
        with:
          python-version: "3.11"
      - name: Setup Terraform
-        uses: hashicorp/setup-terraform@5e8dbf3c6d9deaf4193ca7a8fb23f2ac83bb6c85 # ratchet:hashicorp/setup-terraform@v4.0.0
+        uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # ratchet:hashicorp/setup-terraform@v3
      - name: Setup node
        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v6
        with: # zizmor: ignore[cache-poisoning]
@@ -38,9 +38,9 @@ jobs:
      - name: Install node dependencies
        working-directory: ./web
        run: npm ci
-      - uses: j178/prek-action@0bb87d7f00b0c99306c8bcb8b8beba1eb581c037 # ratchet:j178/prek-action@v1
+      - uses: j178/prek-action@9d6a3097e0c1865ecce00cfb89fe80f2ee91b547 # ratchet:j178/prek-action@v1
        with:
-          prek-version: '0.3.4'
+          prek-version: '0.2.21'
          extra-args: ${{ github.event_name == 'pull_request' && format('--from-ref {0} --to-ref {1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha) || github.event_name == 'merge_group' && format('--from-ref {0} --to-ref {1}', github.event.merge_group.base_sha, github.event.merge_group.head_sha) || github.ref_name == 'main' && '--all-files' || '' }}
      - name: Check Actions
        uses: giner/check-actions@28d366c7cbbe235f9624a88aa31a628167eee28c # ratchet:giner/check-actions@v1.0.1
--- a/.github/workflows/release-cli.yml
+++ b/.github/workflows/release-cli.yml
@@ -1,214 +0,0 @@
-name: Release CLI
-
-on:
-  push:
-    tags:
-      - "cli/v*.*.*"
-
-jobs:
-  pypi:
-    runs-on: ubuntu-latest
-    environment:
-      name: release-cli
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        os-arch:
-          - { goos: "linux", goarch: "amd64" }
-          - { goos: "linux", goarch: "arm64" }
-          - { goos: "windows", goarch: "amd64" }
-          - { goos: "windows", goarch: "arm64" }
-          - { goos: "darwin", goarch: "amd64" }
-          - { goos: "darwin", goarch: "arm64" }
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          enable-cache: false
-          version: "0.9.9"
-      - run: |
-          GOOS="${{ matrix.os-arch.goos }}" \
-          GOARCH="${{ matrix.os-arch.goarch }}" \
-          uv build --wheel
-        working-directory: cli
-      - run: uv publish
-        working-directory: cli
-
-  docker-amd64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-amd64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push AMD64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/amd64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  docker-arm64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-arm64
-      - run-id=${{ github.run_id }}-cli-arm64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push ARM64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/arm64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  merge-docker:
-    needs:
-      - docker-amd64
-      - docker-arm64
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-merge
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Create and push manifest
-        env:
-          AMD64_DIGEST: ${{ needs.docker-amd64.outputs.digest }}
-          ARM64_DIGEST: ${{ needs.docker-arm64.outputs.digest }}
-          TAG: ${{ github.ref_name }}
-        run: |
-          SANITIZED_TAG="${TAG#cli/}"
-          IMAGES=(
-            "${REGISTRY_IMAGE}@${AMD64_DIGEST}"
-            "${REGISTRY_IMAGE}@${ARM64_DIGEST}"
-          )
-
-          if [[ "$TAG" =~ ^cli/v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              -t "${REGISTRY_IMAGE}:latest" \
-              "${IMAGES[@]}"
-          else
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              "${IMAGES[@]}"
-          fi
--- a/.github/workflows/release-devtools.yml
+++ b/.github/workflows/release-devtools.yml
@@ -22,11 +22,13 @@ jobs:
          - { goos: "windows", goarch: "arm64" }
          - { goos: "darwin", goarch: "amd64" }
          - { goos: "darwin", goarch: "arm64" }
+          - { goos: "", goarch: "" }
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
        with:
          persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+          fetch-depth: 0
+      - uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
        with:
          enable-cache: false
          version: "0.9.9"
--- a/.github/workflows/reusable-nightly-llm-provider-chat.yml
+++ b/.github/workflows/reusable-nightly-llm-provider-chat.yml
@@ -49,13 +49,27 @@ on:
        default: true
        type: boolean
    secrets:
-      AWS_OIDC_ROLE_ARN:
-        description: "AWS role ARN for OIDC auth"
+      openai_api_key:
+        required: false
+      anthropic_api_key:
+        required: false
+      bedrock_api_key:
+        required: false
+      vertex_ai_custom_config_json:
+        required: false
+      azure_api_key:
+        required: false
+      ollama_api_key:
+        required: false
+      openrouter_api_key:
+        required: false
+      DOCKER_USERNAME:
+        required: true
+      DOCKER_TOKEN:
        required: true

 permissions:
  contents: read
-  id-token: write

 jobs:
  build-backend-image:
@@ -67,7 +81,6 @@ jobs:
        "extras=ecr-cache",
      ]
    timeout-minutes: 45
-    environment: ci-protected
    steps:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

@@ -76,19 +89,6 @@ jobs:
        with:
          persist-credentials: false

-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
      - name: Build backend image
        uses: ./.github/actions/build-backend-image
        with:
@@ -97,8 +97,8 @@ jobs:
          pr-number: ${{ github.event.pull_request.number }}
          github-sha: ${{ github.sha }}
          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
+          docker-username: ${{ secrets.DOCKER_USERNAME }}
+          docker-token: ${{ secrets.DOCKER_TOKEN }}
          docker-no-cache: ${{ vars.DOCKER_NO_CACHE == 'true' && 'true' || 'false' }}

  build-model-server-image:
@@ -110,7 +110,6 @@ jobs:
        "extras=ecr-cache",
      ]
    timeout-minutes: 45
-    environment: ci-protected
    steps:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

@@ -119,19 +118,6 @@ jobs:
        with:
          persist-credentials: false

-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
      - name: Build model server image
        uses: ./.github/actions/build-model-server-image
        with:
@@ -140,8 +126,8 @@ jobs:
          pr-number: ${{ github.event.pull_request.number }}
          github-sha: ${{ github.sha }}
          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
+          docker-username: ${{ secrets.DOCKER_USERNAME }}
+          docker-token: ${{ secrets.DOCKER_TOKEN }}

  build-integration-image:
    runs-on:
@@ -152,7 +138,6 @@ jobs:
        "extras=ecr-cache",
      ]
    timeout-minutes: 45
-    environment: ci-protected
    steps:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

@@ -161,19 +146,6 @@ jobs:
        with:
          persist-credentials: false

-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
      - name: Build integration image
        uses: ./.github/actions/build-integration-image
        with:
@@ -182,8 +154,8 @@ jobs:
          pr-number: ${{ github.event.pull_request.number }}
          github-sha: ${{ github.sha }}
          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
+          docker-username: ${{ secrets.DOCKER_USERNAME }}
+          docker-token: ${{ secrets.DOCKER_TOKEN }}

  provider-chat-test:
    needs:
@@ -198,56 +170,56 @@ jobs:
        include:
          - provider: openai
            models: ${{ inputs.openai_models }}
-            api_key_env: OPENAI_API_KEY
-            custom_config_env: ""
+            api_key_secret: openai_api_key
+            custom_config_secret: ""
            api_base: ""
            api_version: ""
            deployment_name: ""
            required: true
          - provider: anthropic
            models: ${{ inputs.anthropic_models }}
-            api_key_env: ANTHROPIC_API_KEY
-            custom_config_env: ""
+            api_key_secret: anthropic_api_key
+            custom_config_secret: ""
            api_base: ""
            api_version: ""
            deployment_name: ""
            required: true
          - provider: bedrock
            models: ${{ inputs.bedrock_models }}
-            api_key_env: BEDROCK_API_KEY
-            custom_config_env: ""
+            api_key_secret: bedrock_api_key
+            custom_config_secret: ""
            api_base: ""
            api_version: ""
            deployment_name: ""
            required: false
          - provider: vertex_ai
            models: ${{ inputs.vertex_ai_models }}
-            api_key_env: ""
-            custom_config_env: NIGHTLY_LLM_VERTEX_AI_CUSTOM_CONFIG_JSON
+            api_key_secret: ""
+            custom_config_secret: vertex_ai_custom_config_json
            api_base: ""
            api_version: ""
            deployment_name: ""
            required: false
          - provider: azure
            models: ${{ inputs.azure_models }}
-            api_key_env: AZURE_API_KEY
-            custom_config_env: ""
+            api_key_secret: azure_api_key
+            custom_config_secret: ""
            api_base: ${{ inputs.azure_api_base }}
            api_version: "2025-04-01-preview"
            deployment_name: ""
            required: false
          - provider: ollama_chat
            models: ${{ inputs.ollama_models }}
-            api_key_env: OLLAMA_API_KEY
-            custom_config_env: ""
+            api_key_secret: ollama_api_key
+            custom_config_secret: ""
            api_base: "https://ollama.com"
            api_version: ""
            deployment_name: ""
            required: false
          - provider: openrouter
            models: ${{ inputs.openrouter_models }}
-            api_key_env: OPENROUTER_API_KEY
-            custom_config_env: ""
+            api_key_secret: openrouter_api_key
+            custom_config_secret: ""
            api_base: "https://openrouter.ai/api/v1"
            api_version: ""
            deployment_name: ""
@@ -258,7 +230,6 @@ jobs:
      - "run-id=${{ github.run_id }}-nightly-${{ matrix.provider }}-provider-chat-test"
      - extras=ecr-cache
    timeout-minutes: 45
-    environment: ci-protected
    steps:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

@@ -267,43 +238,21 @@ jobs:
        with:
          persist-credentials: false

-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          # Keep JSON values unparsed so vertex custom config is passed as raw JSON.
-          parse-json-secrets: false
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-            OPENAI_API_KEY, test/openai-api-key
-            ANTHROPIC_API_KEY, test/anthropic-api-key
-            BEDROCK_API_KEY, test/bedrock-api-key
-            NIGHTLY_LLM_VERTEX_AI_CUSTOM_CONFIG_JSON, test/nightly-llm-vertex-ai-custom-config-json
-            AZURE_API_KEY, test/azure-api-key
-            OLLAMA_API_KEY, test/ollama-api-key
-            OPENROUTER_API_KEY, test/openrouter-api-key
-
      - name: Run nightly provider chat test
        uses: ./.github/actions/run-nightly-provider-chat-test
        with:
          provider: ${{ matrix.provider }}
          models: ${{ matrix.models }}
-          provider-api-key: ${{ matrix.api_key_env && env[matrix.api_key_env] || '' }}
+          provider-api-key: ${{ matrix.api_key_secret && secrets[matrix.api_key_secret] || '' }}
          strict: ${{ inputs.strict && 'true' || 'false' }}
          api-base: ${{ matrix.api_base }}
          api-version: ${{ matrix.api_version }}
          deployment-name: ${{ matrix.deployment_name }}
-          custom-config-json: ${{ matrix.custom_config_env && env[matrix.custom_config_env] || '' }}
+          custom-config-json: ${{ matrix.custom_config_secret && secrets[matrix.custom_config_secret] || '' }}
          runs-on-ecr-cache: ${{ env.RUNS_ON_ECR_CACHE }}
          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
+          docker-username: ${{ secrets.DOCKER_USERNAME }}
+          docker-token: ${{ secrets.DOCKER_TOKEN }}

      - name: Dump API server logs
        if: always()
--- a/.github/workflows/sandbox-deployment.yml
+++ b/.github/workflows/sandbox-deployment.yml
@@ -110,7 +110,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -180,7 +180,7 @@ jobs:
          persist-credentials: false

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
@@ -244,7 +244,7 @@ jobs:
      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2

      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
        with:
          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
          aws-region: us-east-2
--- a/.github/workflows/zizmor.yml
+++ b/.github/workflows/zizmor.yml
@@ -24,7 +24,7 @@ jobs:
          persist-credentials: false

      - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
        with:
          enable-cache: false
          version: "0.9.9"
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -119,11 +119,10 @@ repos:
          ]

  - repo: https://github.com/golangci/golangci-lint
-    rev: 5d1e709b7be35cb2025444e19de266b056b7b7ee # frozen: v2.10.1
+    rev: 9f61b0f53f80672872fced07b6874397c3ed197b # frozen: v2.7.2
    hooks:
      - id: golangci-lint
-        language_version: "1.26.0"
-        entry: bash -c "find . -name go.mod -not -path './.venv/*' -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"
+        entry: bash -c "find tools/ -name go.mod -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"

  - repo: https://github.com/astral-sh/ruff-pre-commit
    # Ruff version.
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -40,7 +40,19 @@
      }
    },
    {
-      "name": "Celery",
+      "name": "Celery (lightweight mode)",
+      "configurations": [
+        "Celery primary",
+        "Celery background",
+        "Celery beat"
+      ],
+      "presentation": {
+        "group": "1"
+      },
+      "stopAll": true
+    },
+    {
+      "name": "Celery (standard mode)",
      "configurations": [
        "Celery primary",
        "Celery light",
@@ -241,6 +253,35 @@
      },
      "consoleTitle": "Celery light Console"
    },
+    {
+      "name": "Celery background",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "celery",
+      "cwd": "${workspaceFolder}/backend",
+      "envFile": "${workspaceFolder}/.vscode/.env",
+      "env": {
+        "LOG_LEVEL": "INFO",
+        "PYTHONUNBUFFERED": "1",
+        "PYTHONPATH": "."
+      },
+      "args": [
+        "-A",
+        "onyx.background.celery.versioned_apps.background",
+        "worker",
+        "--pool=threads",
+        "--concurrency=20",
+        "--prefetch-multiplier=4",
+        "--loglevel=INFO",
+        "--hostname=background@%n",
+        "-Q",
+        "vespa_metadata_sync,connector_deletion,doc_permissions_upsert,checkpoint_cleanup,index_attempt_cleanup,docprocessing,connector_doc_fetching,connector_pruning,connector_doc_permissions_sync,connector_external_group_sync,csv_generation,kg_processing,monitoring,user_file_processing,user_file_project_sync,user_file_delete,opensearch_migration"
+      ],
+      "presentation": {
+        "group": "2"
+      },
+      "consoleTitle": "Celery background Console"
+    },
    {
      "name": "Celery heavy",
      "type": "debugpy",
@@ -485,6 +526,21 @@
        "group": "3"
      }
    },
+    {
+      "name": "Clear and Restart OpenSearch Container",
+      // Generic debugger type, required arg but has no bearing on bash.
+      "type": "node",
+      "request": "launch",
+      "runtimeExecutable": "bash",
+      "runtimeArgs": [
+        "${workspaceFolder}/backend/scripts/restart_opensearch_container.sh"
+      ],
+      "cwd": "${workspaceFolder}",
+      "console": "integratedTerminal",
+      "presentation": {
+        "group": "3"
+      }
+    },
    {
      "name": "Eval CLI",
      "type": "debugpy",
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -86,6 +86,37 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work
     - Monitoring tasks (every 5 minutes)
     - Cleanup tasks (hourly)

+#### Worker Deployment Modes
+
+Onyx supports two deployment modes for background workers, controlled by the `USE_LIGHTWEIGHT_BACKGROUND_WORKER` environment variable:
+
+**Lightweight Mode** (default, `USE_LIGHTWEIGHT_BACKGROUND_WORKER=true`):
+
+- Runs a single consolidated `background` worker that handles all background tasks:
+  - Light worker tasks (Vespa operations, permissions sync, deletion)
+  - Document processing (indexing pipeline)
+  - Document fetching (connector data retrieval)
+  - Pruning operations (from `heavy` worker)
+  - Knowledge graph processing (from `kg_processing` worker)
+  - Monitoring tasks (from `monitoring` worker)
+  - User file processing (from `user_file_processing` worker)
+- Lower resource footprint (fewer worker processes)
+- Suitable for smaller deployments or development environments
+- Default concurrency: 20 threads (increased to handle combined workload)
+
+**Standard Mode** (`USE_LIGHTWEIGHT_BACKGROUND_WORKER=false`):
+
+- Runs separate specialized workers as documented above (light, docprocessing, docfetching, heavy, kg_processing, monitoring, user_file_processing)
+- Better isolation and scalability
+- Can scale individual workers independently based on workload
+- Suitable for production deployments with higher load
+
+The deployment mode affects:
+
+- **Backend**: Worker processes spawned by supervisord or dev scripts
+- **Helm**: Which Kubernetes deployments are created
+- **Dev Environment**: Which workers `dev_run_background_jobs.py` spawns
+
 #### Key Features

 - **Thread-based Workers**: All workers use thread pools (not processes) for stability
@@ -104,10 +135,6 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work

 - Always use `@shared_task` rather than `@celery_app`
 - Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks`
- Never enqueue a task without an expiration. Always supply `expires=` when
-  sending tasks, either from the beat schedule or directly from another task. It
-  should never be acceptable to submit code which enqueues tasks without an
-  expiration, as doing so can lead to unbounded task queue growth.

 **Defining APIs**:
 When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the
@@ -544,8 +571,6 @@ To run them:
 npx playwright test <TEST_NAME>
 ```

-For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`.
-
 ## Logs

 When (1) writing integration tests or (2) doing live tests (e.g. curl / playwright) you can get access
@@ -592,45 +617,6 @@ Keep it high level. You can reference certain files or functions though.

 Before writing your plan, make sure to do research. Explore the relevant sections in the codebase.

-## Error Handling
-
-**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`.
-Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.**
-
-A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard
-`{"error_code": "...", "message": "..."}` shape. This eliminates boilerplate and keeps error
-handling consistent across the entire backend.
-
-```python
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
-
-# ✅ Good
-raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found")
-
-# ✅ Good — no extra message needed
-raise OnyxError(OnyxErrorCode.UNAUTHENTICATED)
-
-# ✅ Good — upstream service with dynamic status code
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status)
-
-# ❌ Bad — using HTTPException directly
-raise HTTPException(status_code=404, detail="Session not found")
-
-# ❌ Bad — starlette constant
-raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
-```
-
-Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error
-category is needed, add it there first — do not invent ad-hoc codes.
-
-**Upstream service errors:** When forwarding errors from an upstream service where the HTTP
-status code is dynamic (comes from the upstream response), use `status_code_override`:
-
-```python
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code)
-```
-
 ## Best Practices

 In addition to the other content in this file, best practices for contributing
--- a/backend/Dockerfile
+++ b/backend/Dockerfile
@@ -141,7 +141,6 @@ COPY --chown=onyx:onyx ./scripts/debugging /app/scripts/debugging
 COPY --chown=onyx:onyx ./scripts/force_delete_connector_by_id.py /app/scripts/force_delete_connector_by_id.py
 COPY --chown=onyx:onyx ./scripts/supervisord_entrypoint.sh /app/scripts/supervisord_entrypoint.sh
 COPY --chown=onyx:onyx ./scripts/setup_craft_templates.sh /app/scripts/setup_craft_templates.sh
-COPY --chown=onyx:onyx ./scripts/reencrypt_secrets.py /app/scripts/reencrypt_secrets.py
 RUN chmod +x /app/scripts/supervisord_entrypoint.sh /app/scripts/setup_craft_templates.sh

 # Run Craft template setup at build time when ENABLE_CRAFT=true
--- a/backend/alembic/versions/2664261bfaab_add_cache_store_table.py
+++ b/backend/alembic/versions/2664261bfaab_add_cache_store_table.py
@@ -1,37 +0,0 @@
-"""add cache_store table
-
-Revision ID: 2664261bfaab
-Revises: 4a1e4b1c89d2
-Create Date: 2026-02-27 00:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-# revision identifiers, used by Alembic.
-revision = "2664261bfaab"
-down_revision = "4a1e4b1c89d2"
-branch_labels: None = None
-depends_on: None = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "cache_store",
-        sa.Column("key", sa.String(), nullable=False),
-        sa.Column("value", sa.LargeBinary(), nullable=True),
-        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
-        sa.PrimaryKeyConstraint("key"),
-    )
-    op.create_index(
-        "ix_cache_store_expires",
-        "cache_store",
-        ["expires_at"],
-        postgresql_where=sa.text("expires_at IS NOT NULL"),
-    )
-
-
-def downgrade() -> None:
-    op.drop_index("ix_cache_store_expires", table_name="cache_store")
-    op.drop_table("cache_store")
--- a/backend/alembic/versions/4a1e4b1c89d2_add_indexing_to_userfilestatus.py
+++ b/backend/alembic/versions/4a1e4b1c89d2_add_indexing_to_userfilestatus.py
@@ -1,51 +0,0 @@
-"""Add INDEXING to UserFileStatus
-
-Revision ID: 4a1e4b1c89d2
-Revises: 6b3b4083c5aa
-Create Date: 2026-02-28 00:00:00.000000
-
-"""
-
-import sqlalchemy as sa
-from alembic import op
-
-revision = "4a1e4b1c89d2"
-down_revision = "6b3b4083c5aa"
-branch_labels = None
-depends_on = None
-
-TABLE = "user_file"
-COLUMN = "status"
-CONSTRAINT_NAME = "ck_user_file_status"
-
-OLD_VALUES = ("PROCESSING", "COMPLETED", "FAILED", "CANCELED", "DELETING")
-NEW_VALUES = ("PROCESSING", "INDEXING", "COMPLETED", "FAILED", "CANCELED", "DELETING")
-
-
-def _drop_status_check_constraint() -> None:
-    """Drop the existing CHECK constraint on user_file.status.
-
-    The constraint name is auto-generated by SQLAlchemy and unknown,
-    so we look it up via the inspector.
-    """
-    inspector = sa.inspect(op.get_bind())
-    for constraint in inspector.get_check_constraints(TABLE):
-        if COLUMN in constraint.get("sqltext", ""):
-            constraint_name = constraint["name"]
-            if constraint_name is not None:
-                op.drop_constraint(constraint_name, TABLE, type_="check")
-
-
-def upgrade() -> None:
-    _drop_status_check_constraint()
-    in_clause = ", ".join(f"'{v}'" for v in NEW_VALUES)
-    op.create_check_constraint(CONSTRAINT_NAME, TABLE, f"{COLUMN} IN ({in_clause})")
-
-
-def downgrade() -> None:
-    op.execute(
-        f"UPDATE {TABLE} SET {COLUMN} = 'PROCESSING' WHERE {COLUMN} = 'INDEXING'"
-    )
-    op.drop_constraint(CONSTRAINT_NAME, TABLE, type_="check")
-    in_clause = ", ".join(f"'{v}'" for v in OLD_VALUES)
-    op.create_check_constraint(CONSTRAINT_NAME, TABLE, f"{COLUMN} IN ({in_clause})")
--- a/backend/alembic/versions/6b3b4083c5aa_persona_cleanup_and_featured.py
+++ b/backend/alembic/versions/6b3b4083c5aa_persona_cleanup_and_featured.py
@@ -1,112 +0,0 @@
-"""persona cleanup and featured
-
-Revision ID: 6b3b4083c5aa
-Revises: 57122d037335
-Create Date: 2026-02-26 12:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "6b3b4083c5aa"
-down_revision = "57122d037335"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Add featured column with nullable=True first
-    op.add_column("persona", sa.Column("featured", sa.Boolean(), nullable=True))
-
-    # Migrate data from is_default_persona to featured
-    op.execute("UPDATE persona SET featured = is_default_persona")
-
-    # Make featured non-nullable with default=False
-    op.alter_column(
-        "persona",
-        "featured",
-        existing_type=sa.Boolean(),
-        nullable=False,
-        server_default=sa.false(),
-    )
-
-    # Drop is_default_persona column
-    op.drop_column("persona", "is_default_persona")
-
-    # Drop unused columns
-    op.drop_column("persona", "num_chunks")
-    op.drop_column("persona", "chunks_above")
-    op.drop_column("persona", "chunks_below")
-    op.drop_column("persona", "llm_relevance_filter")
-    op.drop_column("persona", "llm_filter_extraction")
-    op.drop_column("persona", "recency_bias")
-
-
-def downgrade() -> None:
-    # Add back recency_bias column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "recency_bias",
-            sa.VARCHAR(),
-            nullable=False,
-            server_default="base_decay",
-        ),
-    )
-
-    # Add back llm_filter_extraction column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "llm_filter_extraction",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Add back llm_relevance_filter column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "llm_relevance_filter",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Add back chunks_below column
-    op.add_column(
-        "persona",
-        sa.Column("chunks_below", sa.Integer(), nullable=False, server_default="0"),
-    )
-
-    # Add back chunks_above column
-    op.add_column(
-        "persona",
-        sa.Column("chunks_above", sa.Integer(), nullable=False, server_default="0"),
-    )
-
-    # Add back num_chunks column
-    op.add_column("persona", sa.Column("num_chunks", sa.Float(), nullable=True))
-
-    # Add back is_default_persona column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "is_default_persona",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Migrate data from featured to is_default_persona
-    op.execute("UPDATE persona SET is_default_persona = featured")
-
-    # Drop featured column
-    op.drop_column("persona", "featured")
--- a/backend/alembic/versions/a3b8d9e2f1c4_make_scim_external_id_nullable.py
+++ b/backend/alembic/versions/a3b8d9e2f1c4_make_scim_external_id_nullable.py
@@ -1,34 +0,0 @@
-"""make scim_user_mapping.external_id nullable
-
-Revision ID: a3b8d9e2f1c4
-Revises: 2664261bfaab
-Create Date: 2026-03-02
-
-"""
-
-from alembic import op
-
-
-# revision identifiers, used by Alembic.
-revision = "a3b8d9e2f1c4"
-down_revision = "2664261bfaab"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=True,
-    )
-
-
-def downgrade() -> None:
-    # Delete any rows where external_id is NULL before re-applying NOT NULL
-    op.execute("DELETE FROM scim_user_mapping WHERE external_id IS NULL")
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=False,
-    )
--- a/backend/alembic_tenants/versions/3b9f09038764_add_read_only_kg_user.py
+++ b/backend/alembic_tenants/versions/3b9f09038764_add_read_only_kg_user.py
@@ -11,6 +11,7 @@ from sqlalchemy import text
 from alembic import op
 from onyx.configs.app_configs import DB_READONLY_PASSWORD
 from onyx.configs.app_configs import DB_READONLY_USER
+from shared_configs.configs import MULTI_TENANT


 # revision identifiers, used by Alembic.
@@ -21,52 +22,59 @@ depends_on = None


 def upgrade() -> None:
-    # Enable pg_trgm extension if not already enabled
-    op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
+    if MULTI_TENANT:

-    # Create the read-only db user if it does not already exist.
-    if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
-        raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+        # Enable pg_trgm extension if not already enabled
+        op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")

-    op.execute(
-        text(
-            f"""
-            DO $$
-            BEGIN
-                -- Check if the read-only user already exists
-                IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                    -- Create the read-only user with the specified password
-                    EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
-                    -- First revoke all privileges to ensure a clean slate
-                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                    -- Grant only the CONNECT privilege to allow the user to connect to the database
-                    -- but not perform any operations without additional specific grants
-                    EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
-                END IF;
-            END
-            $$;
-            """
+        # Create read-only db user here only in multi-tenant mode. For single-tenant mode,
+        # the user is created in the standard migration.
+        if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
+            raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+
+        op.execute(
+            text(
+                f"""
+                DO $$
+                BEGIN
+                    -- Check if the read-only user already exists
+                    IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                        -- Create the read-only user with the specified password
+                        EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
+                        -- First revoke all privileges to ensure a clean slate
+                        EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                        -- Grant only the CONNECT privilege to allow the user to connect to the database
+                        -- but not perform any operations without additional specific grants
+                        EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
+                    END IF;
+                END
+                $$;
+                """
+            )
        )
-    )


 def downgrade() -> None:
-    op.execute(
-        text(
-            f"""
-        DO $$
-        BEGIN
-            IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                -- First revoke all privileges from the database
-                EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                -- Then revoke all privileges from the public schema
-                EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
-                -- Then drop the user
-                EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
-            END IF;
-        END
-        $$;
-    """
+    if MULTI_TENANT:
+        # Drop read-only db user here only in single tenant mode. For multi-tenant mode,
+        # the user is dropped in the alembic_tenants migration.
+
+        op.execute(
+            text(
+                f"""
+            DO $$
+            BEGIN
+                IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                    -- First revoke all privileges from the database
+                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                    -- Then revoke all privileges from the public schema
+                    EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
+                    -- Then drop the user
+                    EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
+                END IF;
+            END
+            $$;
+        """
+            )
        )
-    )
-    op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))
+        op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))
--- a/backend/ee/onyx/background/celery/apps/background.py
+++ b/backend/ee/onyx/background/celery/apps/background.py
@@ -0,0 +1,15 @@
+from onyx.background.celery.apps import app_base
+from onyx.background.celery.apps.background import celery_app
+
+
+celery_app.autodiscover_tasks(
+    app_base.filter_task_modules(
+        [
+            "ee.onyx.background.celery.tasks.doc_permission_syncing",
+            "ee.onyx.background.celery.tasks.external_group_syncing",
+            "ee.onyx.background.celery.tasks.cleanup",
+            "ee.onyx.background.celery.tasks.tenant_provisioning",
+            "ee.onyx.background.celery.tasks.query_history",
+        ]
+    )
+)
--- a/backend/ee/onyx/db/license.py
+++ b/backend/ee/onyx/db/license.py
@@ -11,10 +11,11 @@ from ee.onyx.server.license.models import LicenseMetadata
 from ee.onyx.server.license.models import LicensePayload
 from ee.onyx.server.license.models import LicenseSource
 from onyx.auth.schemas import UserRole
-from onyx.cache.factory import get_cache_backend
 from onyx.configs.constants import ANONYMOUS_USER_EMAIL
 from onyx.db.models import License
 from onyx.db.models import User
+from onyx.redis.redis_pool import get_redis_client
+from onyx.redis.redis_pool import get_redis_replica_client
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.contextvars import get_current_tenant_id
@@ -141,7 +142,7 @@ def get_used_seats(tenant_id: str | None = None) -> int:

 def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata | None:
    """
-    Get license metadata from cache.
+    Get license metadata from Redis cache.

    Args:
        tenant_id: Tenant ID (for multi-tenant deployments)
@@ -149,34 +150,38 @@ def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata
    Returns:
        LicenseMetadata if cached, None otherwise
    """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cached = cache.get(LICENSE_METADATA_KEY)
-    if not cached:
-        return None
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_replica_client(tenant_id=tenant)

-    try:
-        cached_str = (
-            cached.decode("utf-8") if isinstance(cached, bytes) else str(cached)
-        )
-        return LicenseMetadata.model_validate_json(cached_str)
-    except Exception as e:
-        logger.warning(f"Failed to parse cached license metadata: {e}")
-        return None
+    cached = redis_client.get(LICENSE_METADATA_KEY)
+    if cached:
+        try:
+            cached_str: str
+            if isinstance(cached, bytes):
+                cached_str = cached.decode("utf-8")
+            else:
+                cached_str = str(cached)
+            return LicenseMetadata.model_validate_json(cached_str)
+        except Exception as e:
+            logger.warning(f"Failed to parse cached license metadata: {e}")
+            return None
+    return None


 def invalidate_license_cache(tenant_id: str | None = None) -> None:
    """
    Invalidate the license metadata cache (not the license itself).

-    Deletes the cached LicenseMetadata. The actual license in the database
-    is not affected. Delete is idempotent — if the key doesn't exist, this
-    is a no-op.
+    This deletes the cached LicenseMetadata from Redis. The actual license
+    in the database is not affected. Redis delete is idempotent - if the
+    key doesn't exist, this is a no-op.

    Args:
        tenant_id: Tenant ID (for multi-tenant deployments)
    """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cache.delete(LICENSE_METADATA_KEY)
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_client(tenant_id=tenant)
+    redis_client.delete(LICENSE_METADATA_KEY)
    logger.info("License cache invalidated")


@@ -187,7 +192,7 @@ def update_license_cache(
    tenant_id: str | None = None,
 ) -> LicenseMetadata:
    """
-    Update the cache with license metadata.
+    Update the Redis cache with license metadata.

    We cache all license statuses (ACTIVE, GRACE_PERIOD, GATED_ACCESS) because:
    1. Frontend needs status to show appropriate UI/banners
@@ -206,7 +211,7 @@ def update_license_cache(
    from ee.onyx.utils.license import get_license_status

    tenant = tenant_id or get_current_tenant_id()
-    cache = get_cache_backend(tenant_id=tenant_id)
+    redis_client = get_redis_client(tenant_id=tenant)

    used_seats = get_used_seats(tenant)
    status = get_license_status(payload, grace_period_end)
@@ -225,7 +230,7 @@ def update_license_cache(
        stripe_subscription_id=payload.stripe_subscription_id,
    )

-    cache.set(
+    redis_client.set(
        LICENSE_METADATA_KEY,
        metadata.model_dump_json(),
        ex=LICENSE_CACHE_TTL_SECONDS,
--- a/backend/ee/onyx/db/scim.py
+++ b/backend/ee/onyx/db/scim.py
@@ -126,16 +126,12 @@ class ScimDAL(DAL):

    def create_user_mapping(
        self,
-        external_id: str | None,
+        external_id: str,
        user_id: UUID,
        scim_username: str | None = None,
        fields: ScimMappingFields | None = None,
    ) -> ScimUserMapping:
-        """Create a SCIM mapping for a user.
-
-        ``external_id`` may be ``None`` when the IdP omits it (RFC 7643
-        allows this). The mapping still marks the user as SCIM-managed.
-        """
+        """Create a mapping between a SCIM externalId and an Onyx user."""
        f = fields or ScimMappingFields()
        mapping = ScimUserMapping(
            external_id=external_id,
@@ -274,13 +270,8 @@ class ScimDAL(DAL):
        Raises:
            ValueError: If the filter uses an unsupported attribute.
        """
-        # Inner-join with ScimUserMapping so only SCIM-managed users appear.
-        # Pre-existing system accounts (anonymous, admin, etc.) are excluded
-        # unless they were explicitly linked via SCIM provisioning.
-        query = (
-            select(User)
-            .join(ScimUserMapping, ScimUserMapping.user_id == User.id)
-            .where(User.role.notin_([UserRole.SLACK_USER, UserRole.EXT_PERM_USER]))
+        query = select(User).where(
+            User.role.notin_([UserRole.SLACK_USER, UserRole.EXT_PERM_USER])
        )

        if scim_filter:
@@ -330,37 +321,34 @@ class ScimDAL(DAL):
        scim_username: str | None = None,
        fields: ScimMappingFields | None = None,
    ) -> None:
-        """Sync the SCIM mapping for a user.
-
-        If a mapping already exists, its fields are updated (including
-        setting ``external_id`` to ``None`` when the IdP omits it).
-        If no mapping exists and ``new_external_id`` is provided, a new
-        mapping is created.  A mapping is never deleted here — SCIM-managed
-        users must retain their mapping to remain visible in ``GET /Users``.
+        """Create, update, or delete the external ID mapping for a user.

        When *fields* is provided, all mapping fields are written
        unconditionally — including ``None`` values — so that a caller can
        clear a previously-set field (e.g. removing a department).
        """
        mapping = self.get_user_mapping_by_user_id(user_id)
-        if mapping:
-            if mapping.external_id != new_external_id:
-                mapping.external_id = new_external_id
-            if scim_username is not None:
-                mapping.scim_username = scim_username
-            if fields is not None:
-                mapping.department = fields.department
-                mapping.manager = fields.manager
-                mapping.given_name = fields.given_name
-                mapping.family_name = fields.family_name
-                mapping.scim_emails_json = fields.scim_emails_json
-        elif new_external_id:
-            self.create_user_mapping(
-                external_id=new_external_id,
-                user_id=user_id,
-                scim_username=scim_username,
-                fields=fields,
-            )
+        if new_external_id:
+            if mapping:
+                if mapping.external_id != new_external_id:
+                    mapping.external_id = new_external_id
+                if scim_username is not None:
+                    mapping.scim_username = scim_username
+                if fields is not None:
+                    mapping.department = fields.department
+                    mapping.manager = fields.manager
+                    mapping.given_name = fields.given_name
+                    mapping.family_name = fields.family_name
+                    mapping.scim_emails_json = fields.scim_emails_json
+            else:
+                self.create_user_mapping(
+                    external_id=new_external_id,
+                    user_id=user_id,
+                    scim_username=scim_username,
+                    fields=fields,
+                )
+        elif mapping:
+            self.delete_user_mapping(mapping.id)

    def _get_user_mappings_batch(
        self, user_ids: list[UUID]
--- a/backend/ee/onyx/db/user_group.py
+++ b/backend/ee/onyx/db/user_group.py
@@ -15,7 +15,6 @@ from sqlalchemy.orm import Session
 from ee.onyx.server.user_group.models import SetCuratorRequest
 from ee.onyx.server.user_group.models import UserGroupCreate
 from ee.onyx.server.user_group.models import UserGroupUpdate
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.db.connector_credential_pair import get_connector_credential_pair_from_id
 from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
@@ -472,9 +471,7 @@ def _add_user_group__cc_pair_relationships__no_commit(

 def insert_user_group(db_session: Session, user_group: UserGroupCreate) -> UserGroup:
    db_user_group = UserGroup(
-        name=user_group.name,
-        time_last_modified_by_user=func.now(),
-        is_up_to_date=DISABLE_VECTOR_DB,
+        name=user_group.name, time_last_modified_by_user=func.now()
    )
    db_session.add(db_user_group)
    db_session.flush()  # give the group an ID
@@ -777,7 +774,8 @@ def update_user_group(
            cc_pair_ids=user_group_update.cc_pair_ids,
        )

-    if cc_pairs_updated and not DISABLE_VECTOR_DB:
+    # only needs to sync with Vespa if the cc_pairs have been updated
+    if cc_pairs_updated:
        db_user_group.is_up_to_date = False

    removed_users = db_session.scalars(
--- a/backend/ee/onyx/external_permissions/google_drive/doc_sync.py
+++ b/backend/ee/onyx/external_permissions/google_drive/doc_sync.py
@@ -68,7 +68,6 @@ def get_external_access_for_raw_gdrive_file(
    company_domain: str,
    retriever_drive_service: GoogleDriveService | None,
    admin_drive_service: GoogleDriveService,
-    fallback_user_email: str,
    add_prefix: bool = False,
 ) -> ExternalAccess:
    """
@@ -80,11 +79,6 @@ def get_external_access_for_raw_gdrive_file(
                set add_prefix to True so group IDs are prefixed with the source type.
                When invoked from doc_sync (permission sync), use the default (False)
                since upsert_document_external_perms handles prefixing.
-    fallback_user_email: When we cannot retrieve any permission info for a file
-                (e.g. externally-owned files where the API returns no permissions
-                and permissions.list returns 403), fall back to granting access
-                to this user. This is typically the impersonated org user whose
-                drive contained the file.
    """
    doc_id = file.get("id")
    if not doc_id:
@@ -123,26 +117,6 @@ def get_external_access_for_raw_gdrive_file(
                [permissions_list, backup_permissions_list]
            )

-    # For externally-owned files, the Drive API may return no permissions
-    # and permissions.list may return 403. In this case, fall back to
-    # granting access to the user who found the file in their drive.
-    # Note, even if other users also have access to this file,
-    # they will not be granted access in Onyx.
-    # We check permissions_list (the final result after all fetch attempts)
-    # rather than the raw fields, because permission_ids may be present
-    # but the actual fetch can still return empty due to a 403.
-    if not permissions_list:
-        logger.info(
-            f"No permission info available for file {doc_id} "
-            f"(likely owned by a user outside of your organization). "
-            f"Falling back to granting access to retriever user: {fallback_user_email}"
-        )
-        return ExternalAccess(
-            external_user_emails={fallback_user_email},
-            external_user_group_ids=set(),
-            is_public=False,
-        )
-
    folder_ids_to_inherit_permissions_from: set[str] = set()
    user_emails: set[str] = set()
    group_emails: set[str] = set()
--- a/backend/ee/onyx/main.py
+++ b/backend/ee/onyx/main.py
@@ -4,6 +4,7 @@ from contextlib import asynccontextmanager
 from fastapi import FastAPI
 from httpx_oauth.clients.google import GoogleOAuth2

+from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
 from ee.onyx.server.analytics.api import router as analytics_router
 from ee.onyx.server.auth_check import check_ee_router_auth
 from ee.onyx.server.billing.api import router as billing_router
@@ -30,7 +31,6 @@ from ee.onyx.server.query_and_chat.query_backend import (
 from ee.onyx.server.query_and_chat.search_backend import router as search_router
 from ee.onyx.server.query_history.api import router as query_history_router
 from ee.onyx.server.reporting.usage_export_api import router as usage_export_router
-from ee.onyx.server.scim.api import register_scim_exception_handlers
 from ee.onyx.server.scim.api import scim_router
 from ee.onyx.server.seeding import seed_db
 from ee.onyx.server.tenants.api import router as tenants_router
@@ -152,9 +152,12 @@ def get_application() -> FastAPI:
    # License management
    include_router_with_global_prefix_prepended(application, license_router)

-    # Unified billing API - always registered in EE.
-    # Each endpoint is protected by the `current_admin_user` dependency (admin auth).
-    include_router_with_global_prefix_prepended(application, billing_router)
+    # Unified billing API - available when license system is enabled
+    # Works for both self-hosted and cloud deployments
+    # TODO(ENG-3533): Once frontend migrates to /admin/billing/*, this becomes the
+    # primary billing API and /tenants/* billing endpoints can be removed
+    if LICENSE_ENFORCEMENT_ENABLED:
+        include_router_with_global_prefix_prepended(application, billing_router)

    if MULTI_TENANT:
        # Tenant management
@@ -164,7 +167,6 @@ def get_application() -> FastAPI:
    # they use their own SCIM bearer token auth).
    # Not behind APP_API_PREFIX because IdPs expect /scim/v2/... directly.
    application.include_router(scim_router)
-    register_scim_exception_handlers(application)

    # Ensure all routes have auth enabled or are explicitly marked as public
    check_ee_router_auth(application)
--- a/backend/ee/onyx/server/billing/api.py
+++ b/backend/ee/onyx/server/billing/api.py
@@ -26,6 +26,7 @@ import asyncio
 import httpx
 from fastapi import APIRouter
 from fastapi import Depends
+from fastapi import HTTPException
 from pydantic import BaseModel
 from sqlalchemy.orm import Session

@@ -41,6 +42,7 @@ from ee.onyx.server.billing.models import SeatUpdateRequest
 from ee.onyx.server.billing.models import SeatUpdateResponse
 from ee.onyx.server.billing.models import StripePublishableKeyResponse
 from ee.onyx.server.billing.models import SubscriptionStatusResponse
+from ee.onyx.server.billing.service import BillingServiceError
 from ee.onyx.server.billing.service import (
    create_checkout_session as create_checkout_service,
 )
@@ -56,8 +58,6 @@ from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_OVERRIDE
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_URL
 from onyx.configs.app_configs import WEB_DOMAIN
 from onyx.db.engine.sql_engine import get_session
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.redis.redis_pool import get_shared_redis_client
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
@@ -169,23 +169,26 @@ async def create_checkout_session(
    if seats is not None:
        used_seats = get_used_seats(tenant_id)
        if seats < used_seats:
-            raise OnyxError(
-                OnyxErrorCode.VALIDATION_ERROR,
-                f"Cannot subscribe with fewer seats than current usage. "
+            raise HTTPException(
+                status_code=400,
+                detail=f"Cannot subscribe with fewer seats than current usage. "
                f"You have {used_seats} active users/integrations but requested {seats} seats.",
            )

    # Build redirect URL for after checkout completion
    redirect_url = f"{WEB_DOMAIN}/admin/billing?checkout=success"

-    return await create_checkout_service(
-        billing_period=billing_period,
-        seats=seats,
-        email=email,
-        license_data=license_data,
-        redirect_url=redirect_url,
-        tenant_id=tenant_id,
-    )
+    try:
+        return await create_checkout_service(
+            billing_period=billing_period,
+            seats=seats,
+            email=email,
+            license_data=license_data,
+            redirect_url=redirect_url,
+            tenant_id=tenant_id,
+        )
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)


@router.post("/create-customer-portal-session")
@@ -203,15 +206,18 @@ async def create_customer_portal_session(

    # Self-hosted requires license
    if not MULTI_TENANT and not license_data:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "No license found")
+        raise HTTPException(status_code=400, detail="No license found")

    return_url = request.return_url if request else f"{WEB_DOMAIN}/admin/billing"

-    return await create_portal_service(
-        license_data=license_data,
-        return_url=return_url,
-        tenant_id=tenant_id,
-    )
+    try:
+        return await create_portal_service(
+            license_data=license_data,
+            return_url=return_url,
+            tenant_id=tenant_id,
+        )
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)


@router.get("/billing-information")
@@ -234,9 +240,9 @@ async def get_billing_information(

    # Check circuit breaker (self-hosted only)
    if _is_billing_circuit_open():
-        raise OnyxError(
-            OnyxErrorCode.SERVICE_UNAVAILABLE,
-            "Stripe connection temporarily disabled. Click 'Connect to Stripe' to retry.",
+        raise HTTPException(
+            status_code=503,
+            detail="Stripe connection temporarily disabled. Click 'Connect to Stripe' to retry.",
        )

    try:
@@ -244,15 +250,11 @@ async def get_billing_information(
            license_data=license_data,
            tenant_id=tenant_id,
        )
-    except OnyxError as e:
+    except BillingServiceError as e:
        # Open circuit breaker on connection failures (self-hosted only)
-        if e.status_code in (
-            OnyxErrorCode.BAD_GATEWAY.status_code,
-            OnyxErrorCode.SERVICE_UNAVAILABLE.status_code,
-            OnyxErrorCode.GATEWAY_TIMEOUT.status_code,
-        ):
+        if e.status_code in (502, 503, 504):
            _open_billing_circuit()
-        raise
+        raise HTTPException(status_code=e.status_code, detail=e.message)


@router.post("/seats/update")
@@ -272,25 +274,31 @@ async def update_seats(

    # Self-hosted requires license
    if not MULTI_TENANT and not license_data:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "No license found")
+        raise HTTPException(status_code=400, detail="No license found")

    # Validate that new seat count is not less than current used seats
    used_seats = get_used_seats(tenant_id)
    if request.new_seat_count < used_seats:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            f"Cannot reduce seats below current usage. "
+        raise HTTPException(
+            status_code=400,
+            detail=f"Cannot reduce seats below current usage. "
            f"You have {used_seats} active users/integrations but requested {request.new_seat_count} seats.",
        )

-    # Note: Don't store license here - the control plane may still be processing
-    # the subscription update. The frontend should call /license/claim after a
-    # short delay to get the freshly generated license.
-    return await update_seat_service(
-        new_seat_count=request.new_seat_count,
-        license_data=license_data,
-        tenant_id=tenant_id,
-    )
+    try:
+        result = await update_seat_service(
+            new_seat_count=request.new_seat_count,
+            license_data=license_data,
+            tenant_id=tenant_id,
+        )
+
+        # Note: Don't store license here - the control plane may still be processing
+        # the subscription update. The frontend should call /license/claim after a
+        # short delay to get the freshly generated license.
+
+        return result
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)


@router.get("/stripe-publishable-key")
@@ -321,18 +329,18 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
        if STRIPE_PUBLISHABLE_KEY_OVERRIDE:
            key = STRIPE_PUBLISHABLE_KEY_OVERRIDE.strip()
            if not key.startswith("pk_"):
-                raise OnyxError(
-                    OnyxErrorCode.INTERNAL_ERROR,
-                    "Invalid Stripe publishable key format",
+                raise HTTPException(
+                    status_code=500,
+                    detail="Invalid Stripe publishable key format",
                )
            _stripe_publishable_key_cache = key
            return StripePublishableKeyResponse(publishable_key=key)

        # Fall back to S3 bucket
        if not STRIPE_PUBLISHABLE_KEY_URL:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Stripe publishable key is not configured",
+            raise HTTPException(
+                status_code=500,
+                detail="Stripe publishable key is not configured",
            )

        try:
@@ -343,17 +351,17 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:

                # Validate key format
                if not key.startswith("pk_"):
-                    raise OnyxError(
-                        OnyxErrorCode.INTERNAL_ERROR,
-                        "Invalid Stripe publishable key format",
+                    raise HTTPException(
+                        status_code=500,
+                        detail="Invalid Stripe publishable key format",
                    )

                _stripe_publishable_key_cache = key
                return StripePublishableKeyResponse(publishable_key=key)
        except httpx.HTTPError:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Failed to fetch Stripe publishable key",
+            raise HTTPException(
+                status_code=500,
+                detail="Failed to fetch Stripe publishable key",
            )


--- a/backend/ee/onyx/server/billing/service.py
+++ b/backend/ee/onyx/server/billing/service.py
@@ -22,8 +22,6 @@ from ee.onyx.server.billing.models import SeatUpdateResponse
 from ee.onyx.server.billing.models import SubscriptionStatusResponse
 from ee.onyx.server.tenants.access import generate_data_plane_token
 from onyx.configs.app_configs import CONTROL_PLANE_API_BASE_URL
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT

@@ -33,6 +31,15 @@ logger = setup_logger()
 _REQUEST_TIMEOUT = 30.0


+class BillingServiceError(Exception):
+    """Exception raised for billing service errors."""
+
+    def __init__(self, message: str, status_code: int = 500):
+        self.message = message
+        self.status_code = status_code
+        super().__init__(self.message)
+
+
 def _get_proxy_headers(license_data: str | None) -> dict[str, str]:
    """Build headers for proxy requests (self-hosted).

@@ -94,7 +101,7 @@ async def _make_billing_request(
        Response JSON as dict

    Raises:
-        OnyxError: If request fails
+        BillingServiceError: If request fails
    """

    base_url = _get_base_url()
@@ -121,17 +128,11 @@ async def _make_billing_request(
        except Exception:
            pass
        logger.error(f"{error_message}: {e.response.status_code} - {detail}")
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY,
-            detail,
-            status_code_override=e.response.status_code,
-        )
+        raise BillingServiceError(detail, e.response.status_code)

    except httpx.RequestError:
        logger.exception("Failed to connect to billing service")
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, "Failed to connect to billing service"
-        )
+        raise BillingServiceError("Failed to connect to billing service", 502)


 async def create_checkout_session(
--- a/backend/ee/onyx/server/enterprise_settings/api.py
+++ b/backend/ee/onyx/server/enterprise_settings/api.py
@@ -223,15 +223,6 @@ def get_active_scim_token(
    token = dal.get_active_token()
    if not token:
        raise HTTPException(status_code=404, detail="No active SCIM token")
-
-    # Derive the IdP domain from the first synced user as a heuristic.
-    idp_domain: str | None = None
-    mappings, _total = dal.list_user_mappings(start_index=1, count=1)
-    if mappings:
-        user = dal.get_user(mappings[0].user_id)
-        if user and "@" in user.email:
-            idp_domain = user.email.rsplit("@", 1)[1]
-
    return ScimTokenResponse(
        id=token.id,
        name=token.name,
@@ -239,7 +230,6 @@ def get_active_scim_token(
        is_active=token.is_active,
        created_at=token.created_at,
        last_used_at=token.last_used_at,
-        idp_domain=idp_domain,
    )


--- a/backend/ee/onyx/server/license/api.py
+++ b/backend/ee/onyx/server/license/api.py
@@ -14,6 +14,7 @@ import requests
 from fastapi import APIRouter
 from fastapi import Depends
 from fastapi import File
+from fastapi import HTTPException
 from fastapi import UploadFile
 from sqlalchemy.orm import Session

@@ -34,8 +35,6 @@ from ee.onyx.server.license.models import SeatUsageResponse
 from ee.onyx.utils.license import verify_license_signature
 from onyx.auth.users import User
 from onyx.db.engine.sql_engine import get_session
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT

@@ -128,9 +127,9 @@ async def claim_license(
    2. Without session_id: Re-claim using existing license for auth
    """
    if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License claiming is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License claiming is only available for self-hosted deployments",
        )

    try:
@@ -147,16 +146,15 @@ async def claim_license(
            # Re-claim using existing license for auth
            metadata = get_license_metadata(db_session)
            if not metadata or not metadata.tenant_id:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    "No license found. Provide session_id after checkout.",
+                raise HTTPException(
+                    status_code=400,
+                    detail="No license found. Provide session_id after checkout.",
                )

            license_row = get_license(db_session)
            if not license_row or not license_row.license_data:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    "No license found in database",
+                raise HTTPException(
+                    status_code=400, detail="No license found in database"
                )

            url = f"{CLOUD_DATA_PLANE_URL}/proxy/license/{metadata.tenant_id}"
@@ -175,7 +173,7 @@ async def claim_license(
        license_data = data.get("license")

        if not license_data:
-            raise OnyxError(OnyxErrorCode.NOT_FOUND, "No license in response")
+            raise HTTPException(status_code=404, detail="No license in response")

        # Verify signature before persisting
        payload = verify_license_signature(license_data)
@@ -201,14 +199,12 @@ async def claim_license(
            detail = error_data.get("detail", detail)
        except Exception:
            pass
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=status_code
-        )
+        raise HTTPException(status_code=status_code, detail=detail)
    except ValueError as e:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, str(e))
+        raise HTTPException(status_code=400, detail=str(e))
    except requests.RequestException:
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, "Failed to connect to license server"
+        raise HTTPException(
+            status_code=502, detail="Failed to connect to license server"
        )


@@ -225,9 +221,9 @@ async def upload_license(
    The license file must be cryptographically signed by Onyx.
    """
    if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License upload is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License upload is only available for self-hosted deployments",
        )

    try:
@@ -238,14 +234,14 @@ async def upload_license(
        # Remove any stray whitespace/newlines from user input
        license_data = license_data.strip()
    except UnicodeDecodeError:
-        raise OnyxError(OnyxErrorCode.INVALID_INPUT, "Invalid license file format")
+        raise HTTPException(status_code=400, detail="Invalid license file format")

    # Verify cryptographic signature - this is the only validation needed
    # The license's tenant_id identifies the customer in control plane, not locally
    try:
        payload = verify_license_signature(license_data)
    except ValueError as e:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, str(e))
+        raise HTTPException(status_code=400, detail=str(e))

    # Persist to DB and update cache
    upsert_license(db_session, license_data)
@@ -301,9 +297,9 @@ async def delete_license(
    Admin only - removes license from database and invalidates cache.
    """
    if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License deletion is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License deletion is only available for self-hosted deployments",
        )

    try:
--- a/backend/ee/onyx/server/middleware/license_enforcement.py
+++ b/backend/ee/onyx/server/middleware/license_enforcement.py
@@ -46,6 +46,7 @@ from fastapi import FastAPI
 from fastapi import Request
 from fastapi import Response
 from fastapi.responses import JSONResponse
+from redis.exceptions import RedisError
 from sqlalchemy.exc import SQLAlchemyError

 from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
@@ -55,7 +56,6 @@ from ee.onyx.configs.license_enforcement_config import (
 )
 from ee.onyx.db.license import get_cached_license_metadata
 from ee.onyx.db.license import refresh_license_cache
-from onyx.cache.interface import CACHE_TRANSIENT_ERRORS
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.server.settings.models import ApplicationStatus
 from shared_configs.contextvars import get_current_tenant_id
@@ -164,9 +164,9 @@ def add_license_enforcement_middleware(
                    "[license_enforcement] No license, allowing community features"
                )
                is_gated = False
-        except CACHE_TRANSIENT_ERRORS as e:
+        except RedisError as e:
            logger.warning(f"Failed to check license metadata: {e}")
-            # Fail open - don't block users due to cache connectivity issues
+            # Fail open - don't block users due to Redis connectivity issues
            is_gated = False

        if is_gated:
--- a/backend/ee/onyx/server/scim/api.py
+++ b/backend/ee/onyx/server/scim/api.py
@@ -15,9 +15,7 @@ from uuid import UUID

 from fastapi import APIRouter
 from fastapi import Depends
-from fastapi import FastAPI
 from fastapi import Query
-from fastapi import Request
 from fastapi import Response
 from fastapi.responses import JSONResponse
 from fastapi_users.password import PasswordHelper
@@ -26,7 +24,6 @@ from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session

 from ee.onyx.db.scim import ScimDAL
-from ee.onyx.server.scim.auth import ScimAuthError
 from ee.onyx.server.scim.auth import verify_scim_token
 from ee.onyx.server.scim.filtering import parse_scim_filter
 from ee.onyx.server.scim.models import SCIM_LIST_RESPONSE_SCHEMA
@@ -80,22 +77,6 @@ scim_router = APIRouter(prefix="/scim/v2", tags=["SCIM"])
 _pw_helper = PasswordHelper()


-def register_scim_exception_handlers(app: FastAPI) -> None:
-    """Register SCIM-specific exception handlers on the FastAPI app.
-
-    Call this after ``app.include_router(scim_router)`` so that auth
-    failures from ``verify_scim_token`` return RFC 7644 §3.12 error
-    envelopes (with ``schemas`` and ``status`` fields) instead of
-    FastAPI's default ``{"detail": "..."}`` format.
-    """
-
-    @app.exception_handler(ScimAuthError)
-    async def _handle_scim_auth_error(
-        _request: Request, exc: ScimAuthError
-    ) -> ScimJSONResponse:
-        return _scim_error_response(exc.status_code, exc.detail)
-
-
 def _get_provider(
    _token: ScimToken = Depends(verify_scim_token),
 ) -> ScimProvider:
@@ -423,63 +404,21 @@ def create_user(

    email = user_resource.userName.strip()

-    # Check for existing user — if they exist but aren't SCIM-managed yet,
-    # link them to the IdP rather than rejecting with 409.
-    external_id: str | None = user_resource.externalId
-    scim_username: str = user_resource.userName.strip()
-    fields: ScimMappingFields = _fields_from_resource(user_resource)
+    # externalId is how the IdP correlates this user on subsequent requests.
+    # Without it, the IdP can't find the user and will try to re-create,
+    # hitting a 409 conflict — so we require it up front.
+    if not user_resource.externalId:
+        return _scim_error_response(400, "externalId is required")

-    existing_user = dal.get_user_by_email(email)
-    if existing_user:
-        existing_mapping = dal.get_user_mapping_by_user_id(existing_user.id)
-        if existing_mapping:
-            return _scim_error_response(409, f"User with email {email} already exists")
-
-        # Adopt pre-existing user into SCIM management.
-        # Reactivating a deactivated user consumes a seat, so enforce the
-        # seat limit the same way replace_user does.
-        if user_resource.active and not existing_user.is_active:
-            seat_error = _check_seat_availability(dal)
-            if seat_error:
-                return _scim_error_response(403, seat_error)
-
-        personal_name = _scim_name_to_str(user_resource.name)
-        dal.update_user(
-            existing_user,
-            is_active=user_resource.active,
-            **({"personal_name": personal_name} if personal_name else {}),
-        )
-
-        try:
-            dal.create_user_mapping(
-                external_id=external_id,
-                user_id=existing_user.id,
-                scim_username=scim_username,
-                fields=fields,
-            )
-            dal.commit()
-        except IntegrityError:
-            dal.rollback()
-            return _scim_error_response(
-                409, f"User with email {email} already has a SCIM mapping"
-            )
-
-        return _scim_resource_response(
-            provider.build_user_resource(
-                existing_user,
-                external_id,
-                scim_username=scim_username,
-                fields=fields,
-            ),
-            status_code=201,
-        )
-
-    # Only enforce seat limit for net-new users — adopting a pre-existing
-    # user doesn't consume a new seat.
+    # Enforce seat limit
    seat_error = _check_seat_availability(dal)
    if seat_error:
        return _scim_error_response(403, seat_error)

+    # Check for existing user
+    if dal.get_user_by_email(email):
+        return _scim_error_response(409, f"User with email {email} already exists")
+
    # Create user with a random password (SCIM users authenticate via IdP)
    personal_name = _scim_name_to_str(user_resource.name)
    user = User(
@@ -497,21 +436,18 @@ def create_user(
        dal.rollback()
        return _scim_error_response(409, f"User with email {email} already exists")

-    # Always create a SCIM mapping so that the user is marked as
-    # SCIM-managed. externalId may be None (RFC 7643 says it's optional).
-    try:
-        dal.create_user_mapping(
-            external_id=external_id,
-            user_id=user.id,
-            scim_username=scim_username,
-            fields=fields,
-        )
-        dal.commit()
-    except IntegrityError:
-        dal.rollback()
-        return _scim_error_response(
-            409, f"User with email {email} already has a SCIM mapping"
-        )
+    # Create SCIM mapping (externalId is validated above, always present)
+    external_id = user_resource.externalId
+    scim_username = user_resource.userName.strip()
+    fields = _fields_from_resource(user_resource)
+    dal.create_user_mapping(
+        external_id=external_id,
+        user_id=user.id,
+        scim_username=scim_username,
+        fields=fields,
+    )
+
+    dal.commit()

    return _scim_resource_response(
        provider.build_user_resource(
--- a/backend/ee/onyx/server/scim/auth.py
+++ b/backend/ee/onyx/server/scim/auth.py
@@ -19,6 +19,7 @@ import hashlib
 import secrets

 from fastapi import Depends
+from fastapi import HTTPException
 from fastapi import Request
 from sqlalchemy.orm import Session

@@ -27,21 +28,6 @@ from onyx.auth.utils import get_hashed_bearer_token_from_request
 from onyx.db.engine.sql_engine import get_session
 from onyx.db.models import ScimToken

-
-class ScimAuthError(Exception):
-    """Raised when SCIM bearer token authentication fails.
-
-    Unlike HTTPException, this carries the status and detail so the SCIM
-    exception handler can wrap them in an RFC 7644 §3.12 error envelope
-    with ``schemas`` and ``status`` fields.
-    """
-
-    def __init__(self, status_code: int, detail: str) -> None:
-        self.status_code = status_code
-        self.detail = detail
-        super().__init__(detail)
-
-
 SCIM_TOKEN_PREFIX = "onyx_scim_"
 SCIM_TOKEN_LENGTH = 48

@@ -96,14 +82,23 @@ def verify_scim_token(
    """
    hashed = _get_hashed_scim_token_from_request(request)
    if not hashed:
-        raise ScimAuthError(401, "Missing or invalid SCIM bearer token")
+        raise HTTPException(
+            status_code=401,
+            detail="Missing or invalid SCIM bearer token",
+        )

    token = dal.get_token_by_hash(hashed)

    if not token:
-        raise ScimAuthError(401, "Invalid SCIM bearer token")
+        raise HTTPException(
+            status_code=401,
+            detail="Invalid SCIM bearer token",
+        )

    if not token.is_active:
-        raise ScimAuthError(401, "SCIM token has been revoked")
+        raise HTTPException(
+            status_code=401,
+            detail="SCIM token has been revoked",
+        )

    return token
--- a/backend/ee/onyx/server/scim/models.py
+++ b/backend/ee/onyx/server/scim/models.py
@@ -365,7 +365,6 @@ class ScimTokenResponse(BaseModel):
    is_active: bool
    created_at: datetime
    last_used_at: datetime | None = None
-    idp_domain: str | None = None


 class ScimTokenCreatedResponse(ScimTokenResponse):
--- a/backend/ee/onyx/server/scim/providers/base.py
+++ b/backend/ee/onyx/server/scim/providers/base.py
@@ -153,31 +153,26 @@ class ScimProvider(ABC):
        self,
        user: User,
        fields: ScimMappingFields,
-    ) -> ScimName:
+    ) -> ScimName | None:
        """Build SCIM name components for the response.

        Round-trips stored ``given_name``/``family_name`` when available (so
        the IdP gets back what it sent). Falls back to splitting
        ``personal_name`` for users provisioned before we stored components.
-        Always returns a ScimName — Okta's spec tests expect ``name``
-        (with ``givenName``/``familyName``) on every user resource.
        Providers may override for custom behavior.
        """
        if fields.given_name is not None or fields.family_name is not None:
            return ScimName(
-                givenName=fields.given_name or "",
-                familyName=fields.family_name or "",
-                formatted=user.personal_name or "",
+                givenName=fields.given_name,
+                familyName=fields.family_name,
+                formatted=user.personal_name,
            )
        if not user.personal_name:
-            # Derive a reasonable name from the email so that SCIM spec tests
-            # see non-empty givenName / familyName for every user resource.
-            local = user.email.split("@")[0] if user.email else ""
-            return ScimName(givenName=local, familyName="", formatted=local)
+            return None
        parts = user.personal_name.split(" ", 1)
        return ScimName(
            givenName=parts[0],
-            familyName=parts[1] if len(parts) > 1 else "",
+            familyName=parts[1] if len(parts) > 1 else None,
            formatted=user.personal_name,
        )

--- a/backend/ee/onyx/server/seeding.py
+++ b/backend/ee/onyx/server/seeding.py
@@ -18,6 +18,7 @@ from ee.onyx.server.enterprise_settings.store import (
    store_settings as store_ee_settings,
 )
 from ee.onyx.server.enterprise_settings.store import upload_logo
+from onyx.context.search.enums import RecencyBiasSetting
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.db.llm import fetch_existing_llm_provider
 from onyx.db.llm import update_default_provider
@@ -160,6 +161,12 @@ def _seed_personas(db_session: Session, personas: list[PersonaUpsertRequest]) ->
                    user=None,  # Seeding is done as admin
                    name=persona.name,
                    description=persona.description,
+                    num_chunks=(
+                        persona.num_chunks if persona.num_chunks is not None else 0.0
+                    ),
+                    llm_relevance_filter=persona.llm_relevance_filter,
+                    llm_filter_extraction=persona.llm_filter_extraction,
+                    recency_bias=RecencyBiasSetting.AUTO,
                    document_set_ids=persona.document_set_ids,
                    llm_model_provider_override=persona.llm_model_provider_override,
                    llm_model_version_override=persona.llm_model_version_override,
@@ -171,7 +178,6 @@ def _seed_personas(db_session: Session, personas: list[PersonaUpsertRequest]) ->
                    system_prompt=persona.system_prompt,
                    task_prompt=persona.task_prompt,
                    datetime_aware=persona.datetime_aware,
-                    featured=persona.featured,
                    commit=False,
                )
            db_session.commit()
--- a/backend/ee/onyx/server/settings/api.py
+++ b/backend/ee/onyx/server/settings/api.py
@@ -6,7 +6,6 @@ from sqlalchemy.exc import SQLAlchemyError
 from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
 from ee.onyx.db.license import get_cached_license_metadata
 from ee.onyx.db.license import refresh_license_cache
-from onyx.cache.interface import CACHE_TRANSIENT_ERRORS
 from onyx.configs.app_configs import ENTERPRISE_EDITION_ENABLED
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.server.settings.models import ApplicationStatus
@@ -126,7 +125,7 @@ def apply_license_status_to_settings(settings: Settings) -> Settings:
                # syncing) means indexed data may need protection.
                settings.application_status = _BLOCKING_STATUS
            settings.ee_features_enabled = False
-    except CACHE_TRANSIENT_ERRORS as e:
+    except RedisError as e:
        logger.warning(f"Failed to check license metadata for settings: {e}")
        # Fail closed - disable EE features if we can't verify license
        settings.ee_features_enabled = False
--- a/backend/ee/onyx/server/tenants/billing_api.py
+++ b/backend/ee/onyx/server/tenants/billing_api.py
@@ -21,6 +21,7 @@ import asyncio
 import httpx
 from fastapi import APIRouter
 from fastapi import Depends
+from fastapi import HTTPException

 from ee.onyx.auth.users import current_admin_user
 from ee.onyx.server.tenants.access import control_plane_dep
@@ -42,8 +43,6 @@ from onyx.auth.users import User
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_OVERRIDE
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_URL
 from onyx.configs.app_configs import WEB_DOMAIN
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
 from shared_configs.contextvars import get_current_tenant_id
@@ -117,14 +116,9 @@ async def create_customer_portal_session(
    try:
        portal_url = fetch_customer_portal_session(tenant_id, return_url)
        return {"stripe_customer_portal_url": portal_url}
-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
        logger.exception("Failed to create customer portal session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create customer portal session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))


@router.post("/create-checkout-session")
@@ -140,14 +134,9 @@ async def create_checkout_session(
    try:
        checkout_url = fetch_stripe_checkout_session(tenant_id, billing_period, seats)
        return {"stripe_checkout_url": checkout_url}
-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
        logger.exception("Failed to create checkout session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create checkout session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))


@router.post("/create-subscription-session")
@@ -158,20 +147,15 @@ async def create_subscription_session(
    try:
        tenant_id = CURRENT_TENANT_ID_CONTEXTVAR.get()
        if not tenant_id:
-            raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "Tenant ID not found")
+            raise HTTPException(status_code=400, detail="Tenant ID not found")

        billing_period = request.billing_period if request else "monthly"
        session_id = fetch_stripe_checkout_session(tenant_id, billing_period)
        return SubscriptionSessionResponse(sessionId=session_id)

-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
        logger.exception("Failed to create subscription session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create subscription session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))


@router.get("/stripe-publishable-key")
@@ -202,18 +186,18 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
        if STRIPE_PUBLISHABLE_KEY_OVERRIDE:
            key = STRIPE_PUBLISHABLE_KEY_OVERRIDE.strip()
            if not key.startswith("pk_"):
-                raise OnyxError(
-                    OnyxErrorCode.INTERNAL_ERROR,
-                    "Invalid Stripe publishable key format",
+                raise HTTPException(
+                    status_code=500,
+                    detail="Invalid Stripe publishable key format",
                )
            _stripe_publishable_key_cache = key
            return StripePublishableKeyResponse(publishable_key=key)

        # Fall back to S3 bucket
        if not STRIPE_PUBLISHABLE_KEY_URL:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Stripe publishable key is not configured",
+            raise HTTPException(
+                status_code=500,
+                detail="Stripe publishable key is not configured",
            )

        try:
@@ -224,15 +208,15 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:

                # Validate key format
                if not key.startswith("pk_"):
-                    raise OnyxError(
-                        OnyxErrorCode.INTERNAL_ERROR,
-                        "Invalid Stripe publishable key format",
+                    raise HTTPException(
+                        status_code=500,
+                        detail="Invalid Stripe publishable key format",
                    )

                _stripe_publishable_key_cache = key
                return StripePublishableKeyResponse(publishable_key=key)
        except httpx.HTTPError:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Failed to fetch Stripe publishable key",
+            raise HTTPException(
+                status_code=500,
+                detail="Failed to fetch Stripe publishable key",
            )
--- a/backend/ee/onyx/server/user_group/api.py
+++ b/backend/ee/onyx/server/user_group/api.py
@@ -5,8 +5,6 @@ from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session

 from ee.onyx.db.user_group import add_users_to_user_group
-from ee.onyx.db.user_group import delete_user_group as db_delete_user_group
-from ee.onyx.db.user_group import fetch_user_group
 from ee.onyx.db.user_group import fetch_user_groups
 from ee.onyx.db.user_group import fetch_user_groups_for_user
 from ee.onyx.db.user_group import insert_user_group
@@ -22,7 +20,6 @@ from ee.onyx.server.user_group.models import UserGroupUpdate
 from onyx.auth.users import current_admin_user
 from onyx.auth.users import current_curator_or_admin_user
 from onyx.auth.users import current_user
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.configs.constants import PUBLIC_API_TAGS
 from onyx.db.engine.sql_engine import get_session
 from onyx.db.models import User
@@ -156,8 +153,3 @@ def delete_user_group(
        prepare_user_group_for_deletion(db_session, user_group_id)
    except ValueError as e:
        raise HTTPException(status_code=404, detail=str(e))
-
-    if DISABLE_VECTOR_DB:
-        user_group = fetch_user_group(db_session, user_group_id)
-        if user_group:
-            db_delete_user_group(db_session, user_group)
--- a/backend/ee/onyx/utils/encryption.py
+++ b/backend/ee/onyx/utils/encryption.py
@@ -14,91 +14,67 @@ from onyx.utils.variable_functionality import fetch_versioned_implementation
 logger = setup_logger()


-@lru_cache(maxsize=2)
+@lru_cache(maxsize=1)
 def _get_trimmed_key(key: str) -> bytes:
    encoded_key = key.encode()
    key_length = len(encoded_key)
    if key_length < 16:
        raise RuntimeError("Invalid ENCRYPTION_KEY_SECRET - too short")
+    elif key_length > 32:
+        key = key[:32]
+    elif key_length not in (16, 24, 32):
+        valid_lengths = [16, 24, 32]
+        key = key[: min(valid_lengths, key=lambda x: abs(x - key_length))]

-    # Trim to the largest valid AES key size that fits
-    valid_lengths = [32, 24, 16]
-    for size in valid_lengths:
-        if key_length >= size:
-            return encoded_key[:size]
-
-    raise AssertionError("unreachable")
+    return encoded_key


-def _encrypt_string(input_str: str, key: str | None = None) -> bytes:
-    effective_key = key if key is not None else ENCRYPTION_KEY_SECRET
-    if not effective_key:
+def _encrypt_string(input_str: str) -> bytes:
+    if not ENCRYPTION_KEY_SECRET:
        return input_str.encode()

-    trimmed = _get_trimmed_key(effective_key)
+    key = _get_trimmed_key(ENCRYPTION_KEY_SECRET)
    iv = urandom(16)
    padder = padding.PKCS7(algorithms.AES.block_size).padder()
    padded_data = padder.update(input_str.encode()) + padder.finalize()

-    cipher = Cipher(algorithms.AES(trimmed), modes.CBC(iv), backend=default_backend())
+    cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
    encryptor = cipher.encryptor()
    encrypted_data = encryptor.update(padded_data) + encryptor.finalize()

    return iv + encrypted_data


-def _decrypt_bytes(input_bytes: bytes, key: str | None = None) -> str:
-    effective_key = key if key is not None else ENCRYPTION_KEY_SECRET
-    if not effective_key:
+def _decrypt_bytes(input_bytes: bytes) -> str:
+    if not ENCRYPTION_KEY_SECRET:
        return input_bytes.decode()

-    trimmed = _get_trimmed_key(effective_key)
-    try:
-        iv = input_bytes[:16]
-        encrypted_data = input_bytes[16:]
+    key = _get_trimmed_key(ENCRYPTION_KEY_SECRET)
+    iv = input_bytes[:16]
+    encrypted_data = input_bytes[16:]

-        cipher = Cipher(
-            algorithms.AES(trimmed), modes.CBC(iv), backend=default_backend()
-        )
-        decryptor = cipher.decryptor()
-        decrypted_padded_data = decryptor.update(encrypted_data) + decryptor.finalize()
+    cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
+    decryptor = cipher.decryptor()
+    decrypted_padded_data = decryptor.update(encrypted_data) + decryptor.finalize()

-        unpadder = padding.PKCS7(algorithms.AES.block_size).unpadder()
-        decrypted_data = unpadder.update(decrypted_padded_data) + unpadder.finalize()
+    unpadder = padding.PKCS7(algorithms.AES.block_size).unpadder()
+    decrypted_data = unpadder.update(decrypted_padded_data) + unpadder.finalize()

-        return decrypted_data.decode()
-    except (ValueError, UnicodeDecodeError):
-        if key is not None:
-            # Explicit key was provided — don't fall back silently
-            raise
-        # Read path: attempt raw UTF-8 decode as a fallback for legacy data.
-        # Does NOT handle data encrypted with a different key — that
-        # ciphertext is not valid UTF-8 and will raise below.
-        logger.warning(
-            "AES decryption failed — falling back to raw decode. "
-            "Run the re-encrypt secrets script to rotate to the current key."
-        )
-        try:
-            return input_bytes.decode()
-        except UnicodeDecodeError:
-            raise ValueError(
-                "Data is not valid UTF-8 — likely encrypted with a different key. "
-                "Run the re-encrypt secrets script to rotate to the current key."
-            ) from None
+    return decrypted_data.decode()


-def encrypt_string_to_bytes(input_str: str, key: str | None = None) -> bytes:
+def encrypt_string_to_bytes(input_str: str) -> bytes:
    versioned_encryption_fn = fetch_versioned_implementation(
        "onyx.utils.encryption", "_encrypt_string"
    )
-    return versioned_encryption_fn(input_str, key=key)
+    return versioned_encryption_fn(input_str)


-def decrypt_bytes_to_string(input_bytes: bytes, key: str | None = None) -> str:
+def decrypt_bytes_to_string(input_bytes: bytes) -> str:
    versioned_decryption_fn = fetch_versioned_implementation(
        "onyx.utils.encryption", "_decrypt_bytes"
    )
-    return versioned_decryption_fn(input_bytes, key=key)
+    return versioned_decryption_fn(input_bytes)


 def test_encryption() -> None:
--- a/backend/model_server/encoders.py
+++ b/backend/model_server/encoders.py
@@ -4,11 +4,10 @@ from typing import Any
 from typing import TYPE_CHECKING

 from fastapi import APIRouter
+from fastapi import HTTPException
 from fastapi import Request

 from model_server.utils import simple_log_function_time
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.enums import EmbedTextType
 from shared_configs.model_server_models import Embedding
@@ -189,7 +188,7 @@ async def process_embed_request(
        )

    if not embed_request.texts:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "No texts to be embedded")
+        raise HTTPException(status_code=400, detail="No texts to be embedded")

    if not all(embed_request.texts):
        raise ValueError("Empty strings are not allowed for embedding.")
@@ -212,12 +211,14 @@ async def process_embed_request(
        )
        return EmbedResponse(embeddings=embeddings)
    except RateLimitError as e:
-        raise OnyxError(OnyxErrorCode.RATE_LIMITED, str(e))
+        raise HTTPException(
+            status_code=429,
+            detail=str(e),
+        )
    except Exception as e:
        logger.exception(
            f"Error during embedding process: provider={embed_request.provider_type} model={embed_request.model_name}"
        )
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            f"Error during embedding process: {e}",
+        raise HTTPException(
+            status_code=500, detail=f"Error during embedding process: {e}"
        )
--- a/backend/onyx/auth/users.py
+++ b/backend/onyx/auth/users.py
@@ -120,6 +120,7 @@ from onyx.db.models import User
 from onyx.db.pat import fetch_user_for_pat
 from onyx.db.users import get_user_by_email
 from onyx.redis.redis_pool import get_async_redis_connection
+from onyx.redis.redis_pool import get_redis_client
 from onyx.server.settings.store import load_settings
 from onyx.server.utils import BasicAuthenticationError
 from onyx.utils.logger import setup_logger
@@ -200,14 +201,13 @@ def user_needs_to_be_verified() -> bool:


 def anonymous_user_enabled(*, tenant_id: str | None = None) -> bool:
-    from onyx.cache.factory import get_cache_backend
-
-    cache = get_cache_backend(tenant_id=tenant_id)
-    value = cache.get(OnyxRedisLocks.ANONYMOUS_USER_ENABLED)
+    redis_client = get_redis_client(tenant_id=tenant_id)
+    value = redis_client.get(OnyxRedisLocks.ANONYMOUS_USER_ENABLED)

    if value is None:
        return False

+    assert isinstance(value, bytes)
    return int(value.decode("utf-8")) == 1


@@ -543,7 +543,7 @@ class UserManager(UUIDIDMixin, BaseUserManager[User, uuid.UUID]):
        result = await db_session.execute(
            select(Persona.id)
            .where(
-                Persona.featured.is_(True),
+                Persona.is_default_persona.is_(True),
                Persona.is_public.is_(True),
                Persona.is_visible.is_(True),
                Persona.deleted.is_(False),
@@ -725,19 +725,11 @@ class UserManager(UUIDIDMixin, BaseUserManager[User, uuid.UUID]):
                        if user_by_session:
                            user = user_by_session

-                # If the user is inactive, check seat availability before
-                # upgrading role — otherwise they'd become an inactive BASIC
-                # user who still can't log in.
-                if not user.is_active:
-                    with get_session_with_current_tenant() as sync_db:
-                        enforce_seat_limit(sync_db)
-
                await self.user_db.update(
                    user,
                    {
                        "is_verified": is_verified_by_default,
                        "role": UserRole.BASIC,
-                        **({"is_active": True} if not user.is_active else {}),
                    },
                )

--- a/backend/onyx/background/celery/apps/background.py
+++ b/backend/onyx/background/celery/apps/background.py
@@ -0,0 +1,142 @@
+from typing import Any
+from typing import cast
+
+from celery import Celery
+from celery import signals
+from celery import Task
+from celery.apps.worker import Worker
+from celery.signals import celeryd_init
+from celery.signals import worker_init
+from celery.signals import worker_process_init
+from celery.signals import worker_ready
+from celery.signals import worker_shutdown
+
+import onyx.background.celery.apps.app_base as app_base
+from onyx.background.celery.celery_utils import httpx_init_vespa_pool
+from onyx.configs.app_configs import MANAGED_VESPA
+from onyx.configs.app_configs import VESPA_CLOUD_CERT_PATH
+from onyx.configs.app_configs import VESPA_CLOUD_KEY_PATH
+from onyx.configs.constants import POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME
+from onyx.db.engine.sql_engine import SqlEngine
+from onyx.utils.logger import setup_logger
+from shared_configs.configs import MULTI_TENANT
+
+
+logger = setup_logger()
+
+celery_app = Celery(__name__)
+celery_app.config_from_object("onyx.background.celery.configs.background")
+celery_app.Task = app_base.TenantAwareTask  # type: ignore [misc]
+
+
+@signals.task_prerun.connect
+def on_task_prerun(
+    sender: Any | None = None,
+    task_id: str | None = None,
+    task: Task | None = None,
+    args: tuple | None = None,
+    kwargs: dict | None = None,
+    **kwds: Any,
+) -> None:
+    app_base.on_task_prerun(sender, task_id, task, args, kwargs, **kwds)
+
+
+@signals.task_postrun.connect
+def on_task_postrun(
+    sender: Any | None = None,
+    task_id: str | None = None,
+    task: Task | None = None,
+    args: tuple | None = None,
+    kwargs: dict | None = None,
+    retval: Any | None = None,
+    state: str | None = None,
+    **kwds: Any,
+) -> None:
+    app_base.on_task_postrun(sender, task_id, task, args, kwargs, retval, state, **kwds)
+
+
+@celeryd_init.connect
+def on_celeryd_init(sender: str, conf: Any = None, **kwargs: Any) -> None:
+    app_base.on_celeryd_init(sender, conf, **kwargs)
+
+
+@worker_init.connect
+def on_worker_init(sender: Worker, **kwargs: Any) -> None:
+    EXTRA_CONCURRENCY = 8  # small extra fudge factor for connection limits
+
+    logger.info("worker_init signal received for consolidated background worker.")
+
+    SqlEngine.set_app_name(POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME)
+    pool_size = cast(int, sender.concurrency)  # type: ignore
+    SqlEngine.init_engine(pool_size=pool_size, max_overflow=EXTRA_CONCURRENCY)
+
+    # Initialize Vespa httpx pool (needed for light worker tasks)
+    if MANAGED_VESPA:
+        httpx_init_vespa_pool(
+            sender.concurrency + EXTRA_CONCURRENCY,  # type: ignore
+            ssl_cert=VESPA_CLOUD_CERT_PATH,
+            ssl_key=VESPA_CLOUD_KEY_PATH,
+        )
+    else:
+        httpx_init_vespa_pool(sender.concurrency + EXTRA_CONCURRENCY)  # type: ignore
+
+    app_base.wait_for_redis(sender, **kwargs)
+    app_base.wait_for_db(sender, **kwargs)
+    app_base.wait_for_vespa_or_shutdown(sender, **kwargs)
+
+    # Less startup checks in multi-tenant case
+    if MULTI_TENANT:
+        return
+
+    app_base.on_secondary_worker_init(sender, **kwargs)
+
+
+@worker_ready.connect
+def on_worker_ready(sender: Any, **kwargs: Any) -> None:
+    app_base.on_worker_ready(sender, **kwargs)
+
+
+@worker_shutdown.connect
+def on_worker_shutdown(sender: Any, **kwargs: Any) -> None:
+    app_base.on_worker_shutdown(sender, **kwargs)
+
+
+@worker_process_init.connect
+def init_worker(**kwargs: Any) -> None:  # noqa: ARG001
+    SqlEngine.reset_engine()
+
+
+@signals.setup_logging.connect
+def on_setup_logging(
+    loglevel: Any, logfile: Any, format: Any, colorize: Any, **kwargs: Any
+) -> None:
+    app_base.on_setup_logging(loglevel, logfile, format, colorize, **kwargs)
+
+
+base_bootsteps = app_base.get_bootsteps()
+for bootstep in base_bootsteps:
+    celery_app.steps["worker"].add(bootstep)
+
+celery_app.autodiscover_tasks(
+    app_base.filter_task_modules(
+        [
+            # Original background worker tasks
+            "onyx.background.celery.tasks.pruning",
+            "onyx.background.celery.tasks.monitoring",
+            "onyx.background.celery.tasks.user_file_processing",
+            "onyx.background.celery.tasks.llm_model_update",
+            # Light worker tasks
+            "onyx.background.celery.tasks.shared",
+            "onyx.background.celery.tasks.vespa",
+            "onyx.background.celery.tasks.connector_deletion",
+            "onyx.background.celery.tasks.doc_permission_syncing",
+            "onyx.background.celery.tasks.opensearch_migration",
+            # Docprocessing worker tasks
+            "onyx.background.celery.tasks.docprocessing",
+            # Docfetching worker tasks
+            "onyx.background.celery.tasks.docfetching",
+            # Sandbox cleanup tasks (isolated in build feature)
+            "onyx.server.features.build.sandbox.tasks",
+        ]
+    )
+)
--- a/backend/onyx/background/celery/celery_utils.py
+++ b/backend/onyx/background/celery/celery_utils.py
@@ -39,13 +39,9 @@ CT = TypeVar("CT", bound=ConnectorCheckpoint)


 class SlimConnectorExtractionResult(BaseModel):
-    """Result of extracting document IDs and hierarchy nodes from a connector.
+    """Result of extracting document IDs and hierarchy nodes from a connector."""

-    raw_id_to_parent maps document ID → parent_hierarchy_raw_node_id (or None).
-    Use raw_id_to_parent.keys() wherever the old set of IDs was needed.
-    """
-
-    raw_id_to_parent: dict[str, str | None]
+    doc_ids: set[str]
    hierarchy_nodes: list[HierarchyNode]


@@ -97,37 +93,30 @@ def _get_failure_id(failure: ConnectorFailure) -> str | None:
    return None


-class BatchResult(BaseModel):
-    raw_id_to_parent: dict[str, str | None]
-    hierarchy_nodes: list[HierarchyNode]
-
-
 def _extract_from_batch(
    doc_list: Sequence[Document | SlimDocument | HierarchyNode | ConnectorFailure],
-) -> BatchResult:
-    """Separate a batch into document IDs (with parent mapping) and hierarchy nodes.
+) -> tuple[set[str], list[HierarchyNode]]:
+    """Separate a batch into document IDs and hierarchy nodes.

    ConnectorFailure items have their failed document/entity IDs added to the
-    ID dict so that failed-to-retrieve documents are not accidentally pruned.
+    ID set so that failed-to-retrieve documents are not accidentally pruned.
    """
-    ids: dict[str, str | None] = {}
+    ids: set[str] = set()
    hierarchy_nodes: list[HierarchyNode] = []
    for item in doc_list:
        if isinstance(item, HierarchyNode):
            hierarchy_nodes.append(item)
-            if item.raw_node_id not in ids:
-                ids[item.raw_node_id] = None
+            ids.add(item.raw_node_id)
        elif isinstance(item, ConnectorFailure):
            failed_id = _get_failure_id(item)
            if failed_id:
-                ids[failed_id] = None
+                ids.add(failed_id)
            logger.warning(
                f"Failed to retrieve document {failed_id}: " f"{item.failure_message}"
            )
        else:
-            parent_raw = getattr(item, "parent_hierarchy_raw_node_id", None)
-            ids[item.id] = parent_raw
-    return BatchResult(raw_id_to_parent=ids, hierarchy_nodes=hierarchy_nodes)
+            ids.add(item.id)
+    return ids, hierarchy_nodes


 def extract_ids_from_runnable_connector(
@@ -143,7 +132,7 @@ def extract_ids_from_runnable_connector(

    Optionally, a callback can be passed to handle the length of each document batch.
    """
-    all_raw_id_to_parent: dict[str, str | None] = {}
+    all_connector_doc_ids: set[str] = set()
    all_hierarchy_nodes: list[HierarchyNode] = []

    # Sequence (covariant) lets all the specific list[...] iterator types unify here
@@ -188,20 +177,15 @@ def extract_ids_from_runnable_connector(
                "extract_ids_from_runnable_connector: Stop signal detected"
            )

-        batch_result = _extract_from_batch(doc_list)
-        batch_ids = batch_result.raw_id_to_parent
-        batch_nodes = batch_result.hierarchy_nodes
-        doc_batch_processing_func(batch_ids)
-        for k, v in batch_ids.items():
-            if v is not None or k not in all_raw_id_to_parent:
-                all_raw_id_to_parent[k] = v
+        batch_ids, batch_nodes = _extract_from_batch(doc_list)
+        all_connector_doc_ids.update(doc_batch_processing_func(batch_ids))
        all_hierarchy_nodes.extend(batch_nodes)

        if callback:
            callback.progress("extract_ids_from_runnable_connector", len(batch_ids))

    return SlimConnectorExtractionResult(
-        raw_id_to_parent=all_raw_id_to_parent,
+        doc_ids=all_connector_doc_ids,
        hierarchy_nodes=all_hierarchy_nodes,
    )

--- a/backend/onyx/background/celery/configs/background.py
+++ b/backend/onyx/background/celery/configs/background.py
@@ -0,0 +1,23 @@
+import onyx.background.celery.configs.base as shared_config
+from onyx.configs.app_configs import CELERY_WORKER_BACKGROUND_CONCURRENCY
+
+broker_url = shared_config.broker_url
+broker_connection_retry_on_startup = shared_config.broker_connection_retry_on_startup
+broker_pool_limit = shared_config.broker_pool_limit
+broker_transport_options = shared_config.broker_transport_options
+
+redis_socket_keepalive = shared_config.redis_socket_keepalive
+redis_retry_on_timeout = shared_config.redis_retry_on_timeout
+redis_backend_health_check_interval = shared_config.redis_backend_health_check_interval
+
+result_backend = shared_config.result_backend
+result_expires = shared_config.result_expires  # 86400 seconds is the default
+
+task_default_priority = shared_config.task_default_priority
+task_acks_late = shared_config.task_acks_late
+
+worker_concurrency = CELERY_WORKER_BACKGROUND_CONCURRENCY
+worker_pool = "threads"
+# Increased from 1 to 4 to handle fast light worker tasks more efficiently
+# This allows the worker to prefetch multiple tasks per thread
+worker_prefetch_multiplier = 4
--- a/backend/onyx/background/celery/tasks/beat_schedule.py
+++ b/backend/onyx/background/celery/tasks/beat_schedule.py
@@ -241,7 +241,8 @@ _VECTOR_DB_BEAT_TASK_NAMES: set[str] = {
    "check-for-index-attempt-cleanup",
    "check-for-doc-permissions-sync",
    "check-for-external-group-sync",
-    "migrate-chunks-from-vespa-to-opensearch",
+    "check-for-documents-for-opensearch-migration",
+    "migrate-documents-from-vespa-to-opensearch",
 }

 if DISABLE_VECTOR_DB:
--- a/backend/onyx/background/celery/tasks/opensearch_migration/tasks.py
+++ b/backend/onyx/background/celery/tasks/opensearch_migration/tasks.py
@@ -30,7 +30,6 @@ from onyx.background.celery.tasks.opensearch_migration.transformer import (
    transform_vespa_chunks_to_opensearch_chunks,
 )
 from onyx.configs.app_configs import ENABLE_OPENSEARCH_INDEXING_FOR_ONYX
-from onyx.configs.app_configs import VESPA_MIGRATION_REQUEST_TIMEOUT_S
 from onyx.configs.constants import OnyxCeleryTask
 from onyx.configs.constants import OnyxRedisLocks
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
@@ -48,7 +47,6 @@ from onyx.document_index.interfaces_new import TenantState
 from onyx.document_index.opensearch.opensearch_document_index import (
    OpenSearchDocumentIndex,
 )
-from onyx.document_index.vespa.shared_utils.utils import get_vespa_http_client
 from onyx.document_index.vespa.vespa_document_index import VespaDocumentIndex
 from onyx.indexing.models import IndexingSetting
 from onyx.redis.redis_pool import get_redis_client
@@ -148,12 +146,7 @@ def migrate_chunks_from_vespa_to_opensearch_task(
            task_logger.error(err_str)
            return False

-        with (
-            get_session_with_current_tenant() as db_session,
-            get_vespa_http_client(
-                timeout=VESPA_MIGRATION_REQUEST_TIMEOUT_S
-            ) as vespa_client,
-        ):
+        with get_session_with_current_tenant() as db_session:
            try_insert_opensearch_tenant_migration_record_with_commit(db_session)
            search_settings = get_current_search_settings(db_session)
            tenant_state = TenantState(tenant_id=tenant_id, multitenant=MULTI_TENANT)
@@ -168,7 +161,6 @@ def migrate_chunks_from_vespa_to_opensearch_task(
                index_name=search_settings.index_name,
                tenant_state=tenant_state,
                large_chunks_enabled=False,
-                httpx_client=vespa_client,
            )

            sanitized_doc_start_time = time.monotonic()
--- a/backend/onyx/background/celery/tasks/pruning/tasks.py
+++ b/backend/onyx/background/celery/tasks/pruning/tasks.py
@@ -29,7 +29,6 @@ from onyx.configs.constants import CELERY_GENERIC_BEAT_LOCK_TIMEOUT
 from onyx.configs.constants import CELERY_PRUNING_LOCK_TIMEOUT
 from onyx.configs.constants import CELERY_TASK_WAIT_FOR_FENCE_TIMEOUT
 from onyx.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
-from onyx.configs.constants import DocumentSource
 from onyx.configs.constants import OnyxCeleryPriority
 from onyx.configs.constants import OnyxCeleryQueues
 from onyx.configs.constants import OnyxCeleryTask
@@ -48,8 +47,6 @@ from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
 from onyx.db.enums import SyncStatus
 from onyx.db.enums import SyncType
-from onyx.db.hierarchy import link_hierarchy_nodes_to_documents
-from onyx.db.hierarchy import update_document_parent_hierarchy_nodes
 from onyx.db.hierarchy import upsert_hierarchy_nodes_batch
 from onyx.db.models import ConnectorCredentialPair
 from onyx.db.sync_record import insert_sync_record
@@ -60,8 +57,6 @@ from onyx.redis.redis_connector_prune import RedisConnectorPrune
 from onyx.redis.redis_connector_prune import RedisConnectorPrunePayload
 from onyx.redis.redis_hierarchy import cache_hierarchy_nodes_batch
 from onyx.redis.redis_hierarchy import ensure_source_node_exists
-from onyx.redis.redis_hierarchy import get_node_id_from_raw_id
-from onyx.redis.redis_hierarchy import get_source_node_id_from_cache
 from onyx.redis.redis_hierarchy import HierarchyNodeCacheEntry
 from onyx.redis.redis_pool import get_redis_client
 from onyx.redis.redis_pool import get_redis_replica_client
@@ -118,38 +113,6 @@ class PruneCallback(IndexingCallbackBase):
        super().progress(tag, amount)


-def _resolve_and_update_document_parents(
-    db_session: Session,
-    redis_client: Redis,
-    source: DocumentSource,
-    raw_id_to_parent: dict[str, str | None],
-) -> None:
-    """Resolve parent_hierarchy_raw_node_id → parent_hierarchy_node_id for
-    each document and bulk-update the DB. Mirrors the resolution logic in
-    run_docfetching.py."""
-    source_node_id = get_source_node_id_from_cache(redis_client, db_session, source)
-
-    resolved: dict[str, int | None] = {}
-    for doc_id, raw_parent_id in raw_id_to_parent.items():
-        if raw_parent_id is None:
-            continue
-        node_id, found = get_node_id_from_raw_id(redis_client, source, raw_parent_id)
-        resolved[doc_id] = node_id if found else source_node_id
-
-    if not resolved:
-        return
-
-    update_document_parent_hierarchy_nodes(
-        db_session=db_session,
-        doc_parent_map=resolved,
-        commit=True,
-    )
-    task_logger.info(
-        f"Pruning: resolved and updated parent hierarchy for "
-        f"{len(resolved)} documents (source={source.value})"
-    )
-
-
 """Jobs / utils for kicking off pruning tasks."""


@@ -572,22 +535,22 @@ def connector_pruning_generator_task(
            extraction_result = extract_ids_from_runnable_connector(
                runnable_connector, callback
            )
-            all_connector_doc_ids = extraction_result.raw_id_to_parent
+            all_connector_doc_ids = extraction_result.doc_ids

            # Process hierarchy nodes (same as docfetching):
            # upsert to Postgres and cache in Redis
-            source = cc_pair.connector.source
-            redis_client = get_redis_client(tenant_id=tenant_id)
-
            if extraction_result.hierarchy_nodes:
                is_connector_public = cc_pair.access_type == AccessType.PUBLIC

-                ensure_source_node_exists(redis_client, db_session, source)
+                redis_client = get_redis_client(tenant_id=tenant_id)
+                ensure_source_node_exists(
+                    redis_client, db_session, cc_pair.connector.source
+                )

                upserted_nodes = upsert_hierarchy_nodes_batch(
                    db_session=db_session,
                    nodes=extraction_result.hierarchy_nodes,
-                    source=source,
+                    source=cc_pair.connector.source,
                    commit=True,
                    is_connector_public=is_connector_public,
                )
@@ -598,7 +561,7 @@ def connector_pruning_generator_task(
                ]
                cache_hierarchy_nodes_batch(
                    redis_client=redis_client,
-                    source=source,
+                    source=cc_pair.connector.source,
                    entries=cache_entries,
                )

@@ -607,26 +570,6 @@ def connector_pruning_generator_task(
                    f"hierarchy nodes for cc_pair={cc_pair_id}"
                )

-            ensure_source_node_exists(redis_client, db_session, source)
-            # Resolve parent_hierarchy_raw_node_id → parent_hierarchy_node_id
-            # and bulk-update documents, mirroring the docfetching resolution
-            _resolve_and_update_document_parents(
-                db_session=db_session,
-                redis_client=redis_client,
-                source=source,
-                raw_id_to_parent=all_connector_doc_ids,
-            )
-
-            # Link hierarchy nodes to documents for sources where pages can be
-            # both hierarchy nodes AND documents (e.g. Notion, Confluence)
-            all_doc_id_list = list(all_connector_doc_ids.keys())
-            link_hierarchy_nodes_to_documents(
-                db_session=db_session,
-                document_ids=all_doc_id_list,
-                source=source,
-                commit=True,
-            )
-
            # a list of docs in our local index
            all_indexed_document_ids = {
                doc.id
@@ -638,9 +581,7 @@ def connector_pruning_generator_task(
            }

            # generate list of docs to remove (no longer in the source)
-            doc_ids_to_remove = list(
-                all_indexed_document_ids - all_connector_doc_ids.keys()
-            )
+            doc_ids_to_remove = list(all_indexed_document_ids - all_connector_doc_ids)

            task_logger.info(
                "Pruning set collected: "
--- a/backend/onyx/background/celery/tasks/user_file_processing/tasks.py
+++ b/backend/onyx/background/celery/tasks/user_file_processing/tasks.py
@@ -414,31 +414,34 @@ def _process_user_file_with_indexing(
        raise RuntimeError(f"Indexing pipeline failed for user file {user_file_id}")


-def process_user_file_impl(
-    *, user_file_id: str, tenant_id: str, redis_locking: bool
+@shared_task(
+    name=OnyxCeleryTask.PROCESS_SINGLE_USER_FILE,
+    bind=True,
+    ignore_result=True,
+)
+def process_single_user_file(
+    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
 ) -> None:
-    """Core implementation for processing a single user file.
-
-    When redis_locking=True, acquires a per-file Redis lock and clears the
-    queued-key guard (Celery path).  When redis_locking=False, skips all Redis
-    operations (BackgroundTask path).
-    """
-    task_logger.info(f"process_user_file_impl - Starting id={user_file_id}")
+    task_logger.info(f"process_single_user_file - Starting id={user_file_id}")
    start = time.monotonic()

-    file_lock: RedisLock | None = None
-    if redis_locking:
-        redis_client = get_redis_client(tenant_id=tenant_id)
-        redis_client.delete(_user_file_queued_key(user_file_id))
-        file_lock = redis_client.lock(
-            _user_file_lock_key(user_file_id),
-            timeout=CELERY_USER_FILE_PROCESSING_LOCK_TIMEOUT,
+    redis_client = get_redis_client(tenant_id=tenant_id)
+
+    # Clear the "queued" guard set by the beat generator so that the next beat
+    # cycle can re-enqueue this file if it is still in PROCESSING state after
+    # this task completes or fails.
+    redis_client.delete(_user_file_queued_key(user_file_id))
+
+    file_lock: RedisLock = redis_client.lock(
+        _user_file_lock_key(user_file_id),
+        timeout=CELERY_USER_FILE_PROCESSING_LOCK_TIMEOUT,
+    )
+
+    if not file_lock.acquire(blocking=False):
+        task_logger.info(
+            f"process_single_user_file - Lock held, skipping user_file_id={user_file_id}"
        )
-        if file_lock is not None and not file_lock.acquire(blocking=False):
-            task_logger.info(
-                f"process_user_file_impl - Lock held, skipping user_file_id={user_file_id}"
-            )
-            return
+        return None

    documents: list[Document] = []
    try:
@@ -446,18 +449,15 @@ def process_user_file_impl(
            uf = db_session.get(UserFile, _as_uuid(user_file_id))
            if not uf:
                task_logger.warning(
-                    f"process_user_file_impl - UserFile not found id={user_file_id}"
+                    f"process_single_user_file - UserFile not found id={user_file_id}"
                )
-                return
+                return None

-            if uf.status not in (
-                UserFileStatus.PROCESSING,
-                UserFileStatus.INDEXING,
-            ):
+            if uf.status != UserFileStatus.PROCESSING:
                task_logger.info(
-                    f"process_user_file_impl - Skipping id={user_file_id} status={uf.status}"
+                    f"process_single_user_file - Skipping id={user_file_id} status={uf.status}"
                )
-                return
+                return None

            connector = LocalFileConnector(
                file_locations=[uf.file_id],
@@ -471,6 +471,7 @@ def process_user_file_impl(
                        [doc for doc in batch if not isinstance(doc, HierarchyNode)]
                    )

+                # update the document id to userfile id in the documents
                for document in documents:
                    document.id = str(user_file_id)
                    document.source = DocumentSource.USER_FILE
@@ -492,8 +493,9 @@ def process_user_file_impl(

            except Exception as e:
                task_logger.exception(
-                    f"process_user_file_impl - Error processing file id={user_file_id} - {e.__class__.__name__}"
+                    f"process_single_user_file - Error processing file id={user_file_id} - {e.__class__.__name__}"
                )
+                # don't update the status if the user file is being deleted
                current_user_file = db_session.get(UserFile, _as_uuid(user_file_id))
                if (
                    current_user_file
@@ -502,43 +504,33 @@ def process_user_file_impl(
                    uf.status = UserFileStatus.FAILED
                    db_session.add(uf)
                    db_session.commit()
-                return
+                return None

        elapsed = time.monotonic() - start
        task_logger.info(
-            f"process_user_file_impl - Finished id={user_file_id} docs={len(documents)} elapsed={elapsed:.2f}s"
+            f"process_single_user_file - Finished id={user_file_id} docs={len(documents)} elapsed={elapsed:.2f}s"
        )
+        return None
    except Exception as e:
+        # Attempt to mark the file as failed
        with get_session_with_current_tenant() as db_session:
            uf = db_session.get(UserFile, _as_uuid(user_file_id))
            if uf:
+                # don't update the status if the user file is being deleted
                if uf.status != UserFileStatus.DELETING:
                    uf.status = UserFileStatus.FAILED
                db_session.add(uf)
                db_session.commit()

        task_logger.exception(
-            f"process_user_file_impl - Error processing file id={user_file_id} - {e.__class__.__name__}"
+            f"process_single_user_file - Error processing file id={user_file_id} - {e.__class__.__name__}"
        )
-        raise
+        return None
    finally:
-        if file_lock is not None and file_lock.owned():
+        if file_lock.owned():
            file_lock.release()


-@shared_task(
-    name=OnyxCeleryTask.PROCESS_SINGLE_USER_FILE,
-    bind=True,
-    ignore_result=True,
-)
-def process_single_user_file(
-    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
-) -> None:
-    process_user_file_impl(
-        user_file_id=user_file_id, tenant_id=tenant_id, redis_locking=True
-    )
-
-
@shared_task(
    name=OnyxCeleryTask.CHECK_FOR_USER_FILE_DELETE,
    soft_time_limit=300,
@@ -589,38 +581,36 @@ def check_for_user_file_delete(self: Task, *, tenant_id: str) -> None:
    return None


-def delete_user_file_impl(
-    *, user_file_id: str, tenant_id: str, redis_locking: bool
+@shared_task(
+    name=OnyxCeleryTask.DELETE_SINGLE_USER_FILE,
+    bind=True,
+    ignore_result=True,
+)
+def process_single_user_file_delete(
+    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
 ) -> None:
-    """Core implementation for deleting a single user file.
-
-    When redis_locking=True, acquires a per-file Redis lock (Celery path).
-    When redis_locking=False, skips Redis operations (BackgroundTask path).
-    """
-    task_logger.info(f"delete_user_file_impl - Starting id={user_file_id}")
-
-    file_lock: RedisLock | None = None
-    if redis_locking:
-        redis_client = get_redis_client(tenant_id=tenant_id)
-        file_lock = redis_client.lock(
-            _user_file_delete_lock_key(user_file_id),
-            timeout=CELERY_GENERIC_BEAT_LOCK_TIMEOUT,
+    """Process a single user file delete."""
+    task_logger.info(f"process_single_user_file_delete - Starting id={user_file_id}")
+    redis_client = get_redis_client(tenant_id=tenant_id)
+    file_lock: RedisLock = redis_client.lock(
+        _user_file_delete_lock_key(user_file_id),
+        timeout=CELERY_GENERIC_BEAT_LOCK_TIMEOUT,
+    )
+    if not file_lock.acquire(blocking=False):
+        task_logger.info(
+            f"process_single_user_file_delete - Lock held, skipping user_file_id={user_file_id}"
        )
-        if file_lock is not None and not file_lock.acquire(blocking=False):
-            task_logger.info(
-                f"delete_user_file_impl - Lock held, skipping user_file_id={user_file_id}"
-            )
-            return
-
+        return None
    try:
        with get_session_with_current_tenant() as db_session:
            user_file = db_session.get(UserFile, _as_uuid(user_file_id))
            if not user_file:
                task_logger.info(
-                    f"delete_user_file_impl - User file not found id={user_file_id}"
+                    f"process_single_user_file_delete - User file not found id={user_file_id}"
                )
-                return
+                return None

+            # 1) Delete vector DB chunks (skip when disabled)
            if not DISABLE_VECTOR_DB:
                if MANAGED_VESPA:
                    httpx_init_vespa_pool(
@@ -658,6 +648,7 @@ def delete_user_file_impl(
                        chunk_count=chunk_count,
                    )

+            # 2) Delete the user-uploaded file content from filestore (blob + metadata)
            file_store = get_default_file_store()
            try:
                file_store.delete_file(user_file.file_id)
@@ -665,34 +656,26 @@ def delete_user_file_impl(
                    user_file_id_to_plaintext_file_name(user_file.id)
                )
            except Exception as e:
+                # This block executed only if the file is not found in the filestore
                task_logger.exception(
-                    f"delete_user_file_impl - Error deleting file id={user_file.id} - {e.__class__.__name__}"
+                    f"process_single_user_file_delete - Error deleting file id={user_file.id} - {e.__class__.__name__}"
                )

+            # 3) Finally, delete the UserFile row
            db_session.delete(user_file)
            db_session.commit()
-            task_logger.info(f"delete_user_file_impl - Completed id={user_file_id}")
+            task_logger.info(
+                f"process_single_user_file_delete - Completed id={user_file_id}"
+            )
    except Exception as e:
        task_logger.exception(
-            f"delete_user_file_impl - Error processing file id={user_file_id} - {e.__class__.__name__}"
+            f"process_single_user_file_delete - Error processing file id={user_file_id} - {e.__class__.__name__}"
        )
-        raise
+        return None
    finally:
-        if file_lock is not None and file_lock.owned():
+        if file_lock.owned():
            file_lock.release()
-
-
-@shared_task(
-    name=OnyxCeleryTask.DELETE_SINGLE_USER_FILE,
-    bind=True,
-    ignore_result=True,
-)
-def process_single_user_file_delete(
-    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
-) -> None:
-    delete_user_file_impl(
-        user_file_id=user_file_id, tenant_id=tenant_id, redis_locking=True
-    )
+    return None


@shared_task(
@@ -764,30 +747,32 @@ def check_for_user_file_project_sync(self: Task, *, tenant_id: str) -> None:
    return None


-def project_sync_user_file_impl(
-    *, user_file_id: str, tenant_id: str, redis_locking: bool
+@shared_task(
+    name=OnyxCeleryTask.PROCESS_SINGLE_USER_FILE_PROJECT_SYNC,
+    bind=True,
+    ignore_result=True,
+)
+def process_single_user_file_project_sync(
+    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
 ) -> None:
-    """Core implementation for syncing a user file's project/persona metadata.
+    """Process a single user file project sync."""
+    task_logger.info(
+        f"process_single_user_file_project_sync - Starting id={user_file_id}"
+    )

-    When redis_locking=True, acquires a per-file Redis lock and clears the
-    queued-key guard (Celery path).  When redis_locking=False, skips Redis
-    operations (BackgroundTask path).
-    """
-    task_logger.info(f"project_sync_user_file_impl - Starting id={user_file_id}")
+    redis_client = get_redis_client(tenant_id=tenant_id)
+    redis_client.delete(_user_file_project_sync_queued_key(user_file_id))

-    file_lock: RedisLock | None = None
-    if redis_locking:
-        redis_client = get_redis_client(tenant_id=tenant_id)
-        redis_client.delete(_user_file_project_sync_queued_key(user_file_id))
-        file_lock = redis_client.lock(
-            user_file_project_sync_lock_key(user_file_id),
-            timeout=CELERY_USER_FILE_PROJECT_SYNC_LOCK_TIMEOUT,
+    file_lock: RedisLock = redis_client.lock(
+        user_file_project_sync_lock_key(user_file_id),
+        timeout=CELERY_USER_FILE_PROJECT_SYNC_LOCK_TIMEOUT,
+    )
+
+    if not file_lock.acquire(blocking=False):
+        task_logger.info(
+            f"process_single_user_file_project_sync - Lock held, skipping user_file_id={user_file_id}"
        )
-        if file_lock is not None and not file_lock.acquire(blocking=False):
-            task_logger.info(
-                f"project_sync_user_file_impl - Lock held, skipping user_file_id={user_file_id}"
-            )
-            return
+        return None

    try:
        with get_session_with_current_tenant() as db_session:
@@ -798,10 +783,11 @@ def project_sync_user_file_impl(
            ).scalar_one_or_none()
            if not user_file:
                task_logger.info(
-                    f"project_sync_user_file_impl - User file not found id={user_file_id}"
+                    f"process_single_user_file_project_sync - User file not found id={user_file_id}"
                )
-                return
+                return None

+            # Sync project metadata to vector DB (skip when disabled)
            if not DISABLE_VECTOR_DB:
                if MANAGED_VESPA:
                    httpx_init_vespa_pool(
@@ -836,7 +822,7 @@ def project_sync_user_file_impl(
                    )

            task_logger.info(
-                f"project_sync_user_file_impl - User file id={user_file_id}"
+                f"process_single_user_file_project_sync - User file id={user_file_id}"
            )

            user_file.needs_project_sync = False
@@ -849,22 +835,11 @@ def project_sync_user_file_impl(

    except Exception as e:
        task_logger.exception(
-            f"project_sync_user_file_impl - Error syncing project for file id={user_file_id} - {e.__class__.__name__}"
+            f"process_single_user_file_project_sync - Error syncing project for file id={user_file_id} - {e.__class__.__name__}"
        )
-        raise
+        return None
    finally:
-        if file_lock is not None and file_lock.owned():
+        if file_lock.owned():
            file_lock.release()

-
-@shared_task(
-    name=OnyxCeleryTask.PROCESS_SINGLE_USER_FILE_PROJECT_SYNC,
-    bind=True,
-    ignore_result=True,
-)
-def process_single_user_file_project_sync(
-    self: Task, *, user_file_id: str, tenant_id: str  # noqa: ARG001
-) -> None:
-    project_sync_user_file_impl(
-        user_file_id=user_file_id, tenant_id=tenant_id, redis_locking=True
-    )
+    return None
--- a/backend/onyx/background/celery/versioned_apps/background.py
+++ b/backend/onyx/background/celery/versioned_apps/background.py
@@ -0,0 +1,10 @@
+from celery import Celery
+
+from onyx.utils.variable_functionality import fetch_versioned_implementation
+from onyx.utils.variable_functionality import set_is_ee_based_on_env_variable
+
+set_is_ee_based_on_env_variable()
+app: Celery = fetch_versioned_implementation(
+    "onyx.background.celery.apps.background",
+    "celery_app",
+)
--- a/backend/onyx/background/indexing/run_docfetching.py
+++ b/backend/onyx/background/indexing/run_docfetching.py
@@ -58,6 +58,8 @@ from onyx.file_store.document_batch_storage import DocumentBatchStorage
 from onyx.file_store.document_batch_storage import get_document_batch_storage
 from onyx.indexing.indexing_heartbeat import IndexingHeartbeatInterface
 from onyx.indexing.indexing_pipeline import index_doc_batch_prepare
+from onyx.indexing.postgres_sanitization import sanitize_document_for_postgres
+from onyx.indexing.postgres_sanitization import sanitize_hierarchy_nodes_for_postgres
 from onyx.redis.redis_hierarchy import cache_hierarchy_nodes_batch
 from onyx.redis.redis_hierarchy import ensure_source_node_exists
 from onyx.redis.redis_hierarchy import get_node_id_from_raw_id
@@ -69,8 +71,6 @@ from onyx.server.features.build.indexing.persistent_document_writer import (
 )
 from onyx.utils.logger import setup_logger
 from onyx.utils.middleware import make_randomized_onyx_request_id
-from onyx.utils.postgres_sanitization import sanitize_document_for_postgres
-from onyx.utils.postgres_sanitization import sanitize_hierarchy_nodes_for_postgres
 from onyx.utils.variable_functionality import global_version
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.contextvars import INDEX_ATTEMPT_INFO_CONTEXTVAR
--- a/backend/onyx/background/periodic_poller.py
+++ b/backend/onyx/background/periodic_poller.py
@@ -1,307 +0,0 @@
-"""Periodic poller for NO_VECTOR_DB deployments.
-
-Replaces Celery Beat and background workers with a lightweight daemon thread
-that runs from the API server process.  Two responsibilities:
-
-1. Recovery polling (every 30 s): re-processes user files stuck in
-   PROCESSING / DELETING / needs_sync states via the drain loops defined
-   in ``task_utils.py``.
-
-2. Periodic task execution (configurable intervals): runs LLM model updates
-   and scheduled evals at their configured cadences, with Postgres advisory
-   lock deduplication across multiple API server instances.
-"""
-
-import threading
-import time
-from collections.abc import Callable
-from dataclasses import dataclass
-from dataclasses import field
-
-from onyx.utils.logger import setup_logger
-
-logger = setup_logger()
-
-RECOVERY_INTERVAL_SECONDS = 30
-PERIODIC_TASK_LOCK_BASE = 20_000
-PERIODIC_TASK_KV_PREFIX = "periodic_poller:last_claimed:"
-
-
-# ------------------------------------------------------------------
-# Periodic task definitions
-# ------------------------------------------------------------------
-
-
-_NEVER_RAN: float = -1e18
-
-
-@dataclass
-class _PeriodicTaskDef:
-    name: str
-    interval_seconds: float
-    lock_id: int
-    run_fn: Callable[[], None]
-    last_run_at: float = field(default=_NEVER_RAN)
-
-
-def _run_auto_llm_update() -> None:
-    from onyx.configs.app_configs import AUTO_LLM_CONFIG_URL
-
-    if not AUTO_LLM_CONFIG_URL:
-        return
-
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-    from onyx.llm.well_known_providers.auto_update_service import (
-        sync_llm_models_from_github,
-    )
-
-    with get_session_with_current_tenant() as db_session:
-        sync_llm_models_from_github(db_session)
-
-
-def _run_cache_cleanup() -> None:
-    from onyx.cache.postgres_backend import cleanup_expired_cache_entries
-
-    cleanup_expired_cache_entries()
-
-
-def _run_scheduled_eval() -> None:
-    from onyx.configs.app_configs import BRAINTRUST_API_KEY
-    from onyx.configs.app_configs import SCHEDULED_EVAL_DATASET_NAMES
-    from onyx.configs.app_configs import SCHEDULED_EVAL_PERMISSIONS_EMAIL
-    from onyx.configs.app_configs import SCHEDULED_EVAL_PROJECT
-
-    if not all(
-        [
-            BRAINTRUST_API_KEY,
-            SCHEDULED_EVAL_PROJECT,
-            SCHEDULED_EVAL_DATASET_NAMES,
-            SCHEDULED_EVAL_PERMISSIONS_EMAIL,
-        ]
-    ):
-        return
-
-    from datetime import datetime
-    from datetime import timezone
-
-    from onyx.evals.eval import run_eval
-    from onyx.evals.models import EvalConfigurationOptions
-
-    run_timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d")
-    for dataset_name in SCHEDULED_EVAL_DATASET_NAMES:
-        try:
-            run_eval(
-                configuration=EvalConfigurationOptions(
-                    search_permissions_email=SCHEDULED_EVAL_PERMISSIONS_EMAIL,
-                    dataset_name=dataset_name,
-                    no_send_logs=False,
-                    braintrust_project=SCHEDULED_EVAL_PROJECT,
-                    experiment_name=f"{dataset_name} - {run_timestamp}",
-                ),
-                remote_dataset_name=dataset_name,
-            )
-        except Exception:
-            logger.exception(
-                f"Periodic poller - Failed scheduled eval for dataset {dataset_name}"
-            )
-
-
-_CACHE_CLEANUP_INTERVAL_SECONDS = 300
-
-
-def _build_periodic_tasks() -> list[_PeriodicTaskDef]:
-    from onyx.cache.interface import CacheBackendType
-    from onyx.configs.app_configs import AUTO_LLM_CONFIG_URL
-    from onyx.configs.app_configs import AUTO_LLM_UPDATE_INTERVAL_SECONDS
-    from onyx.configs.app_configs import CACHE_BACKEND
-    from onyx.configs.app_configs import SCHEDULED_EVAL_DATASET_NAMES
-
-    tasks: list[_PeriodicTaskDef] = []
-    if CACHE_BACKEND == CacheBackendType.POSTGRES:
-        tasks.append(
-            _PeriodicTaskDef(
-                name="cache-cleanup",
-                interval_seconds=_CACHE_CLEANUP_INTERVAL_SECONDS,
-                lock_id=PERIODIC_TASK_LOCK_BASE + 2,
-                run_fn=_run_cache_cleanup,
-            )
-        )
-    if AUTO_LLM_CONFIG_URL:
-        tasks.append(
-            _PeriodicTaskDef(
-                name="auto-llm-update",
-                interval_seconds=AUTO_LLM_UPDATE_INTERVAL_SECONDS,
-                lock_id=PERIODIC_TASK_LOCK_BASE,
-                run_fn=_run_auto_llm_update,
-            )
-        )
-    if SCHEDULED_EVAL_DATASET_NAMES:
-        tasks.append(
-            _PeriodicTaskDef(
-                name="scheduled-eval",
-                interval_seconds=7 * 24 * 3600,
-                lock_id=PERIODIC_TASK_LOCK_BASE + 1,
-                run_fn=_run_scheduled_eval,
-            )
-        )
-    return tasks
-
-
-# ------------------------------------------------------------------
-# Periodic task runner with advisory-lock-guarded claim
-# ------------------------------------------------------------------
-
-
-def _try_claim_task(task_def: _PeriodicTaskDef) -> bool:
-    """Atomically check whether *task_def* should run and record a claim.
-
-    Uses a transaction-scoped advisory lock for atomicity combined with a
-    ``KVStore`` timestamp for cross-instance dedup.  The DB session is held
-    only for this brief claim transaction, not during task execution.
-    """
-    from datetime import datetime
-    from datetime import timezone
-
-    from sqlalchemy import text
-
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-    from onyx.db.models import KVStore
-
-    kv_key = PERIODIC_TASK_KV_PREFIX + task_def.name
-
-    with get_session_with_current_tenant() as db_session:
-        acquired = db_session.execute(
-            text("SELECT pg_try_advisory_xact_lock(:id)"),
-            {"id": task_def.lock_id},
-        ).scalar()
-        if not acquired:
-            return False
-
-        row = db_session.query(KVStore).filter_by(key=kv_key).first()
-        if row and row.value is not None:
-            last_claimed = datetime.fromisoformat(str(row.value))
-            elapsed = (datetime.now(timezone.utc) - last_claimed).total_seconds()
-            if elapsed < task_def.interval_seconds:
-                return False
-
-        now_ts = datetime.now(timezone.utc).isoformat()
-        if row:
-            row.value = now_ts
-        else:
-            db_session.add(KVStore(key=kv_key, value=now_ts))
-        db_session.commit()
-
-    return True
-
-
-def _try_run_periodic_task(task_def: _PeriodicTaskDef) -> None:
-    """Run *task_def* if its interval has elapsed and no peer holds the lock."""
-    now = time.monotonic()
-    if now - task_def.last_run_at < task_def.interval_seconds:
-        return
-
-    if not _try_claim_task(task_def):
-        return
-
-    try:
-        task_def.run_fn()
-        task_def.last_run_at = now
-    except Exception:
-        logger.exception(
-            f"Periodic poller - Error running periodic task {task_def.name}"
-        )
-
-
-# ------------------------------------------------------------------
-# Recovery / drain loop runner
-# ------------------------------------------------------------------
-
-
-def _run_drain_loops(tenant_id: str) -> None:
-    from onyx.background.task_utils import drain_delete_loop
-    from onyx.background.task_utils import drain_processing_loop
-    from onyx.background.task_utils import drain_project_sync_loop
-
-    drain_processing_loop(tenant_id)
-    drain_delete_loop(tenant_id)
-    drain_project_sync_loop(tenant_id)
-
-
-# ------------------------------------------------------------------
-# Startup recovery (10g)
-# ------------------------------------------------------------------
-
-
-def recover_stuck_user_files(tenant_id: str) -> None:
-    """Run all drain loops once to re-process files left in intermediate states.
-
-    Called from ``lifespan()`` on startup when ``DISABLE_VECTOR_DB`` is set.
-    """
-    logger.info("recover_stuck_user_files - Checking for stuck user files")
-    try:
-        _run_drain_loops(tenant_id)
-    except Exception:
-        logger.exception("recover_stuck_user_files - Error during recovery")
-
-
-# ------------------------------------------------------------------
-# Daemon thread (10f)
-# ------------------------------------------------------------------
-
-_shutdown_event = threading.Event()
-_poller_thread: threading.Thread | None = None
-
-
-def _poller_loop(tenant_id: str) -> None:
-    from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
-
-    CURRENT_TENANT_ID_CONTEXTVAR.set(tenant_id)
-
-    periodic_tasks = _build_periodic_tasks()
-    logger.info(
-        f"Periodic poller started with {len(periodic_tasks)} periodic task(s): "
-        f"{[t.name for t in periodic_tasks]}"
-    )
-
-    while not _shutdown_event.is_set():
-        try:
-            _run_drain_loops(tenant_id)
-        except Exception:
-            logger.exception("Periodic poller - Error in recovery polling")
-
-        for task_def in periodic_tasks:
-            try:
-                _try_run_periodic_task(task_def)
-            except Exception:
-                logger.exception(
-                    f"Periodic poller - Unhandled error checking task {task_def.name}"
-                )
-
-        _shutdown_event.wait(RECOVERY_INTERVAL_SECONDS)
-
-
-def start_periodic_poller(tenant_id: str) -> None:
-    """Start the periodic poller daemon thread."""
-    global _poller_thread  # noqa: PLW0603
-    _shutdown_event.clear()
-    _poller_thread = threading.Thread(
-        target=_poller_loop,
-        args=(tenant_id,),
-        daemon=True,
-        name="no-vectordb-periodic-poller",
-    )
-    _poller_thread.start()
-    logger.info("Periodic poller thread started")
-
-
-def stop_periodic_poller() -> None:
-    """Signal the periodic poller to stop and wait for it to exit."""
-    global _poller_thread  # noqa: PLW0603
-    if _poller_thread is None:
-        return
-    _shutdown_event.set()
-    _poller_thread.join(timeout=10)
-    if _poller_thread.is_alive():
-        logger.warning("Periodic poller thread did not stop within timeout")
-    _poller_thread = None
-    logger.info("Periodic poller thread stopped")
--- a/backend/onyx/background/task_utils.py
+++ b/backend/onyx/background/task_utils.py
@@ -1,33 +1,3 @@
-"""Background task utilities.
-
-Contains query-history report helpers (used by all deployment modes) and
-in-process background task execution helpers for NO_VECTOR_DB mode:
-
- Atomic claim-and-mark helpers that prevent duplicate processing
- Drain loops that process all pending user file work
-
-Each claim function runs a short-lived transaction: SELECT ... FOR UPDATE
-SKIP LOCKED, UPDATE the row to remove it from future queries, COMMIT.
-After the commit the row lock is released, but the row is no longer
-eligible for re-claiming.  No long-lived sessions or advisory locks.
-"""
-
-from uuid import UUID
-
-import sqlalchemy as sa
-from sqlalchemy import select
-from sqlalchemy.orm import Session
-
-from onyx.db.enums import UserFileStatus
-from onyx.db.models import UserFile
-from onyx.utils.logger import setup_logger
-
-logger = setup_logger()
-
-# ------------------------------------------------------------------
-# Query-history report helpers (pre-existing, used by all modes)
-# ------------------------------------------------------------------
-
 QUERY_REPORT_NAME_PREFIX = "query-history"


@@ -39,168 +9,3 @@ def construct_query_history_report_name(

 def extract_task_id_from_query_history_report_name(name: str) -> str:
    return name.removeprefix(f"{QUERY_REPORT_NAME_PREFIX}-").removesuffix(".csv")
-
-
-# ------------------------------------------------------------------
-# Atomic claim-and-mark helpers
-# ------------------------------------------------------------------
-# Each function runs inside a single short-lived session/transaction:
-#   1. SELECT ... FOR UPDATE SKIP LOCKED  (locks one eligible row)
-#   2. UPDATE the row so it is no longer eligible
-#   3. COMMIT  (releases the row lock)
-# After the commit, no other drain loop can claim the same row.
-
-
-def _claim_next_processing_file(db_session: Session) -> UUID | None:
-    """Claim the next PROCESSING file by transitioning it to INDEXING.
-
-    Returns the file id, or None when no eligible files remain.
-    """
-    file_id = db_session.execute(
-        select(UserFile.id)
-        .where(UserFile.status == UserFileStatus.PROCESSING)
-        .order_by(UserFile.created_at)
-        .limit(1)
-        .with_for_update(skip_locked=True)
-    ).scalar_one_or_none()
-    if file_id is None:
-        return None
-
-    db_session.execute(
-        sa.update(UserFile)
-        .where(UserFile.id == file_id)
-        .values(status=UserFileStatus.INDEXING)
-    )
-    db_session.commit()
-    return file_id
-
-
-def _claim_next_deleting_file(
-    db_session: Session,
-    exclude_ids: set[UUID] | None = None,
-) -> UUID | None:
-    """Claim the next DELETING file.
-
-    No status transition needed — the impl deletes the row on success.
-    The short-lived FOR UPDATE lock prevents concurrent claims.
-    *exclude_ids* prevents re-processing the same file if the impl fails.
-    """
-    stmt = (
-        select(UserFile.id)
-        .where(UserFile.status == UserFileStatus.DELETING)
-        .order_by(UserFile.created_at)
-        .limit(1)
-        .with_for_update(skip_locked=True)
-    )
-    if exclude_ids:
-        stmt = stmt.where(UserFile.id.notin_(exclude_ids))
-    file_id = db_session.execute(stmt).scalar_one_or_none()
-    db_session.commit()
-    return file_id
-
-
-def _claim_next_sync_file(
-    db_session: Session,
-    exclude_ids: set[UUID] | None = None,
-) -> UUID | None:
-    """Claim the next file needing project/persona sync.
-
-    No status transition needed — the impl clears the sync flags on
-    success.  The short-lived FOR UPDATE lock prevents concurrent claims.
-    *exclude_ids* prevents re-processing the same file if the impl fails.
-    """
-    stmt = (
-        select(UserFile.id)
-        .where(
-            sa.and_(
-                sa.or_(
-                    UserFile.needs_project_sync.is_(True),
-                    UserFile.needs_persona_sync.is_(True),
-                ),
-                UserFile.status == UserFileStatus.COMPLETED,
-            )
-        )
-        .order_by(UserFile.created_at)
-        .limit(1)
-        .with_for_update(skip_locked=True)
-    )
-    if exclude_ids:
-        stmt = stmt.where(UserFile.id.notin_(exclude_ids))
-    file_id = db_session.execute(stmt).scalar_one_or_none()
-    db_session.commit()
-    return file_id
-
-
-# ------------------------------------------------------------------
-# Drain loops — process *all* pending work of each type
-# ------------------------------------------------------------------
-
-
-def drain_processing_loop(tenant_id: str) -> None:
-    """Process all pending PROCESSING user files."""
-    from onyx.background.celery.tasks.user_file_processing.tasks import (
-        process_user_file_impl,
-    )
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-
-    while True:
-        with get_session_with_current_tenant() as session:
-            file_id = _claim_next_processing_file(session)
-        if file_id is None:
-            break
-        try:
-            process_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to process user file {file_id}")
-
-
-def drain_delete_loop(tenant_id: str) -> None:
-    """Delete all pending DELETING user files."""
-    from onyx.background.celery.tasks.user_file_processing.tasks import (
-        delete_user_file_impl,
-    )
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-
-    failed: set[UUID] = set()
-    while True:
-        with get_session_with_current_tenant() as session:
-            file_id = _claim_next_deleting_file(session, exclude_ids=failed)
-        if file_id is None:
-            break
-        try:
-            delete_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to delete user file {file_id}")
-            failed.add(file_id)
-
-
-def drain_project_sync_loop(tenant_id: str) -> None:
-    """Sync all pending project/persona metadata for user files."""
-    from onyx.background.celery.tasks.user_file_processing.tasks import (
-        project_sync_user_file_impl,
-    )
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-
-    failed: set[UUID] = set()
-    while True:
-        with get_session_with_current_tenant() as session:
-            file_id = _claim_next_sync_file(session, exclude_ids=failed)
-        if file_id is None:
-            break
-        try:
-            project_sync_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to sync user file {file_id}")
-            failed.add(file_id)
--- a/backend/onyx/cache/factory.py
+++ b/backend/onyx/cache/factory.py
@@ -1,51 +0,0 @@
-from collections.abc import Callable
-
-from onyx.cache.interface import CacheBackend
-from onyx.cache.interface import CacheBackendType
-from onyx.configs.app_configs import CACHE_BACKEND
-
-
-def _build_redis_backend(tenant_id: str) -> CacheBackend:
-    from onyx.cache.redis_backend import RedisCacheBackend
-    from onyx.redis.redis_pool import redis_pool
-
-    return RedisCacheBackend(redis_pool.get_client(tenant_id))
-
-
-def _build_postgres_backend(tenant_id: str) -> CacheBackend:
-    from onyx.cache.postgres_backend import PostgresCacheBackend
-
-    return PostgresCacheBackend(tenant_id)
-
-
-_BACKEND_BUILDERS: dict[CacheBackendType, Callable[[str], CacheBackend]] = {
-    CacheBackendType.REDIS: _build_redis_backend,
-    CacheBackendType.POSTGRES: _build_postgres_backend,
-}
-
-
-def get_cache_backend(*, tenant_id: str | None = None) -> CacheBackend:
-    """Return a tenant-aware ``CacheBackend``.
-
-    If *tenant_id* is ``None``, the current tenant is read from the
-    thread-local context variable (same behaviour as ``get_redis_client``).
-    """
-    if tenant_id is None:
-        from shared_configs.contextvars import get_current_tenant_id
-
-        tenant_id = get_current_tenant_id()
-
-    builder = _BACKEND_BUILDERS.get(CACHE_BACKEND)
-    if builder is None:
-        raise ValueError(
-            f"Unsupported CACHE_BACKEND={CACHE_BACKEND!r}. "
-            f"Supported values: {[t.value for t in CacheBackendType]}"
-        )
-    return builder(tenant_id)
-
-
-def get_shared_cache_backend() -> CacheBackend:
-    """Return a ``CacheBackend`` in the shared (cross-tenant) namespace."""
-    from shared_configs.configs import DEFAULT_REDIS_PREFIX
-
-    return get_cache_backend(tenant_id=DEFAULT_REDIS_PREFIX)
--- a/backend/onyx/cache/interface.py
+++ b/backend/onyx/cache/interface.py
@@ -1,115 +0,0 @@
-import abc
-from enum import Enum
-
-from redis.exceptions import RedisError
-from sqlalchemy.exc import SQLAlchemyError
-
-TTL_KEY_NOT_FOUND = -2
-TTL_NO_EXPIRY = -1
-
-CACHE_TRANSIENT_ERRORS: tuple[type[Exception], ...] = (RedisError, SQLAlchemyError)
-"""Exception types that represent transient cache connectivity / operational
-failures.  Callers that want to fail-open (or fail-closed) on cache errors
-should catch this tuple instead of bare ``Exception``.
-
-When adding a new ``CacheBackend`` implementation, add its transient error
-base class(es) here so all call-sites pick it up automatically."""
-
-
-class CacheBackendType(str, Enum):
-    REDIS = "redis"
-    POSTGRES = "postgres"
-
-
-class CacheLock(abc.ABC):
-    """Abstract distributed lock returned by CacheBackend.lock()."""
-
-    @abc.abstractmethod
-    def acquire(
-        self,
-        blocking: bool = True,
-        blocking_timeout: float | None = None,
-    ) -> bool:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def release(self) -> None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def owned(self) -> bool:
-        raise NotImplementedError
-
-    def __enter__(self) -> "CacheLock":
-        if not self.acquire():
-            raise RuntimeError("Failed to acquire lock")
-        return self
-
-    def __exit__(self, *args: object) -> None:
-        self.release()
-
-
-class CacheBackend(abc.ABC):
-    """Thin abstraction over a key-value cache with TTL, locks, and blocking lists.
-
-    Covers the subset of Redis operations used outside of Celery. When
-    CACHE_BACKEND=postgres, a PostgreSQL-backed implementation is used instead.
-    """
-
-    # -- basic key/value ---------------------------------------------------
-
-    @abc.abstractmethod
-    def get(self, key: str) -> bytes | None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def set(
-        self,
-        key: str,
-        value: str | bytes | int | float,
-        ex: int | None = None,
-    ) -> None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def delete(self, key: str) -> None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def exists(self, key: str) -> bool:
-        raise NotImplementedError
-
-    # -- TTL ---------------------------------------------------------------
-
-    @abc.abstractmethod
-    def expire(self, key: str, seconds: int) -> None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def ttl(self, key: str) -> int:
-        """Return remaining TTL in seconds.
-
-        Returns ``TTL_NO_EXPIRY`` (-1) if key exists without expiry,
-        ``TTL_KEY_NOT_FOUND`` (-2) if key is missing or expired.
-        """
-        raise NotImplementedError
-
-    # -- distributed lock --------------------------------------------------
-
-    @abc.abstractmethod
-    def lock(self, name: str, timeout: float | None = None) -> CacheLock:
-        raise NotImplementedError
-
-    # -- blocking list (used by MCP OAuth BLPOP pattern) -------------------
-
-    @abc.abstractmethod
-    def rpush(self, key: str, value: str | bytes) -> None:
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def blpop(self, keys: list[str], timeout: int = 0) -> tuple[bytes, bytes] | None:
-        """Block until a value is available on one of *keys*, or *timeout* expires.
-
-        Returns ``(key, value)`` or ``None`` on timeout.
-        """
-        raise NotImplementedError
--- a/backend/onyx/cache/postgres_backend.py
+++ b/backend/onyx/cache/postgres_backend.py
@@ -1,323 +0,0 @@
-"""PostgreSQL-backed ``CacheBackend`` for NO_VECTOR_DB deployments.
-
-Uses the ``cache_store`` table for key-value storage, PostgreSQL advisory locks
-for distributed locking, and a polling loop for the BLPOP pattern.
-"""
-
-import hashlib
-import struct
-import time
-import uuid
-from contextlib import AbstractContextManager
-from datetime import datetime
-from datetime import timedelta
-from datetime import timezone
-
-from sqlalchemy import delete
-from sqlalchemy import func
-from sqlalchemy import or_
-from sqlalchemy import select
-from sqlalchemy import update
-from sqlalchemy.dialects.postgresql import insert as pg_insert
-from sqlalchemy.orm import Session
-
-from onyx.cache.interface import CacheBackend
-from onyx.cache.interface import CacheLock
-from onyx.cache.interface import TTL_KEY_NOT_FOUND
-from onyx.cache.interface import TTL_NO_EXPIRY
-from onyx.db.models import CacheStore
-
-_LIST_KEY_PREFIX = "_q:"
-# ASCII: ':' (0x3A) < ';' (0x3B). Upper bound for range queries so [prefix+, prefix;)
-# captures all list-item keys (e.g. _q:mylist:123:uuid) without including other
-# lists whose names share a prefix (e.g. _q:mylist2:...).
-_LIST_KEY_RANGE_TERMINATOR = ";"
-_LIST_ITEM_TTL_SECONDS = 3600
-_LOCK_POLL_INTERVAL = 0.1
-_BLPOP_POLL_INTERVAL = 0.25
-
-
-def _list_item_key(key: str) -> str:
-    """Unique key for a list item. Timestamp for FIFO ordering; UUID prevents
-    collision when concurrent rpush calls occur within the same nanosecond.
-    """
-    return f"{_LIST_KEY_PREFIX}{key}:{time.time_ns()}:{uuid.uuid4().hex}"
-
-
-def _to_bytes(value: str | bytes | int | float) -> bytes:
-    if isinstance(value, bytes):
-        return value
-    return str(value).encode()
-
-
-# ------------------------------------------------------------------
-# Lock
-# ------------------------------------------------------------------
-
-
-class PostgresCacheLock(CacheLock):
-    """Advisory-lock-based distributed lock.
-
-    Uses ``get_session_with_tenant`` for connection lifecycle.  The lock is tied
-    to the session's connection; releasing or closing the session frees it.
-
-    NOTE: Unlike Redis locks, advisory locks do not auto-expire after
-    ``timeout`` seconds.  They are released when ``release()`` is
-    called or when the session is closed.
-    """
-
-    def __init__(self, lock_id: int, timeout: float | None, tenant_id: str) -> None:
-        self._lock_id = lock_id
-        self._timeout = timeout
-        self._tenant_id = tenant_id
-        self._session_cm: AbstractContextManager[Session] | None = None
-        self._session: Session | None = None
-        self._acquired = False
-
-    def acquire(
-        self,
-        blocking: bool = True,
-        blocking_timeout: float | None = None,
-    ) -> bool:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        self._session_cm = get_session_with_tenant(tenant_id=self._tenant_id)
-        self._session = self._session_cm.__enter__()
-        try:
-            if not blocking:
-                return self._try_lock()
-
-            effective_timeout = blocking_timeout or self._timeout
-            deadline = (
-                (time.monotonic() + effective_timeout) if effective_timeout else None
-            )
-            while True:
-                if self._try_lock():
-                    return True
-                if deadline is not None and time.monotonic() >= deadline:
-                    return False
-                time.sleep(_LOCK_POLL_INTERVAL)
-        finally:
-            if not self._acquired:
-                self._close_session()
-
-    def release(self) -> None:
-        if not self._acquired or self._session is None:
-            return
-        try:
-            self._session.execute(select(func.pg_advisory_unlock(self._lock_id)))
-        finally:
-            self._acquired = False
-            self._close_session()
-
-    def owned(self) -> bool:
-        return self._acquired
-
-    def _close_session(self) -> None:
-        if self._session_cm is not None:
-            try:
-                self._session_cm.__exit__(None, None, None)
-            finally:
-                self._session_cm = None
-                self._session = None
-
-    def _try_lock(self) -> bool:
-        assert self._session is not None
-        result = self._session.execute(
-            select(func.pg_try_advisory_lock(self._lock_id))
-        ).scalar()
-        if result:
-            self._acquired = True
-            return True
-        return False
-
-
-# ------------------------------------------------------------------
-# Backend
-# ------------------------------------------------------------------
-
-
-class PostgresCacheBackend(CacheBackend):
-    """``CacheBackend`` backed by the ``cache_store`` table in PostgreSQL.
-
-    Each operation opens and closes its own database session so the backend
-    is safe to share across threads.  Tenant isolation is handled by
-    SQLAlchemy's ``schema_translate_map`` (set by ``get_session_with_tenant``).
-    """
-
-    def __init__(self, tenant_id: str) -> None:
-        self._tenant_id = tenant_id
-
-    # -- basic key/value ---------------------------------------------------
-
-    def get(self, key: str) -> bytes | None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = select(CacheStore.value).where(
-            CacheStore.key == key,
-            or_(CacheStore.expires_at.is_(None), CacheStore.expires_at > func.now()),
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            value = session.execute(stmt).scalar_one_or_none()
-        if value is None:
-            return None
-        return bytes(value)
-
-    def set(
-        self,
-        key: str,
-        value: str | bytes | int | float,
-        ex: int | None = None,
-    ) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        value_bytes = _to_bytes(value)
-        expires_at = (
-            datetime.now(timezone.utc) + timedelta(seconds=ex)
-            if ex is not None
-            else None
-        )
-        stmt = (
-            pg_insert(CacheStore)
-            .values(key=key, value=value_bytes, expires_at=expires_at)
-            .on_conflict_do_update(
-                index_elements=[CacheStore.key],
-                set_={"value": value_bytes, "expires_at": expires_at},
-            )
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(stmt)
-            session.commit()
-
-    def delete(self, key: str) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(delete(CacheStore).where(CacheStore.key == key))
-            session.commit()
-
-    def exists(self, key: str) -> bool:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = (
-            select(CacheStore.key)
-            .where(
-                CacheStore.key == key,
-                or_(
-                    CacheStore.expires_at.is_(None),
-                    CacheStore.expires_at > func.now(),
-                ),
-            )
-            .limit(1)
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            return session.execute(stmt).first() is not None
-
-    # -- TTL ---------------------------------------------------------------
-
-    def expire(self, key: str, seconds: int) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        new_exp = datetime.now(timezone.utc) + timedelta(seconds=seconds)
-        stmt = (
-            update(CacheStore).where(CacheStore.key == key).values(expires_at=new_exp)
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(stmt)
-            session.commit()
-
-    def ttl(self, key: str) -> int:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = select(CacheStore.expires_at).where(CacheStore.key == key)
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            result = session.execute(stmt).first()
-        if result is None:
-            return TTL_KEY_NOT_FOUND
-        expires_at: datetime | None = result[0]
-        if expires_at is None:
-            return TTL_NO_EXPIRY
-        remaining = (expires_at - datetime.now(timezone.utc)).total_seconds()
-        if remaining <= 0:
-            return TTL_KEY_NOT_FOUND
-        return int(remaining)
-
-    # -- distributed lock --------------------------------------------------
-
-    def lock(self, name: str, timeout: float | None = None) -> CacheLock:
-        return PostgresCacheLock(
-            self._lock_id_for(name), timeout, tenant_id=self._tenant_id
-        )
-
-    # -- blocking list (MCP OAuth BLPOP pattern) ---------------------------
-
-    def rpush(self, key: str, value: str | bytes) -> None:
-        self.set(_list_item_key(key), value, ex=_LIST_ITEM_TTL_SECONDS)
-
-    def blpop(self, keys: list[str], timeout: int = 0) -> tuple[bytes, bytes] | None:
-        if timeout <= 0:
-            raise ValueError(
-                "PostgresCacheBackend.blpop requires timeout > 0. "
-                "timeout=0 would block the calling thread indefinitely "
-                "with no way to interrupt short of process termination."
-            )
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        deadline = time.monotonic() + timeout
-        while True:
-            for key in keys:
-                lower = f"{_LIST_KEY_PREFIX}{key}:"
-                upper = f"{_LIST_KEY_PREFIX}{key}{_LIST_KEY_RANGE_TERMINATOR}"
-                stmt = (
-                    select(CacheStore)
-                    .where(
-                        CacheStore.key >= lower,
-                        CacheStore.key < upper,
-                        or_(
-                            CacheStore.expires_at.is_(None),
-                            CacheStore.expires_at > func.now(),
-                        ),
-                    )
-                    .order_by(CacheStore.key)
-                    .limit(1)
-                    .with_for_update(skip_locked=True)
-                )
-                with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-                    row = session.execute(stmt).scalars().first()
-                    if row is not None:
-                        value = bytes(row.value) if row.value else b""
-                        session.delete(row)
-                        session.commit()
-                        return (key.encode(), value)
-            if time.monotonic() >= deadline:
-                return None
-            time.sleep(_BLPOP_POLL_INTERVAL)
-
-    # -- helpers -----------------------------------------------------------
-
-    def _lock_id_for(self, name: str) -> int:
-        """Map *name* to a 64-bit signed int for ``pg_advisory_lock``."""
-        h = hashlib.md5(f"{self._tenant_id}:{name}".encode()).digest()
-        return struct.unpack("q", h[:8])[0]
-
-
-# ------------------------------------------------------------------
-# Periodic cleanup
-# ------------------------------------------------------------------
-
-
-def cleanup_expired_cache_entries() -> None:
-    """Delete rows whose ``expires_at`` is in the past.
-
-    Called by the periodic poller every 5 minutes.
-    """
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-
-    with get_session_with_current_tenant() as session:
-        session.execute(
-            delete(CacheStore).where(
-                CacheStore.expires_at.is_not(None),
-                CacheStore.expires_at < func.now(),
-            )
-        )
-        session.commit()
--- a/backend/onyx/cache/redis_backend.py
+++ b/backend/onyx/cache/redis_backend.py
@@ -1,92 +0,0 @@
-from typing import cast
-
-from redis.client import Redis
-from redis.lock import Lock as RedisLock
-
-from onyx.cache.interface import CacheBackend
-from onyx.cache.interface import CacheLock
-
-
-class RedisCacheLock(CacheLock):
-    """Wraps ``redis.lock.Lock`` behind the ``CacheLock`` interface."""
-
-    def __init__(self, lock: RedisLock) -> None:
-        self._lock = lock
-
-    def acquire(
-        self,
-        blocking: bool = True,
-        blocking_timeout: float | None = None,
-    ) -> bool:
-        return bool(
-            self._lock.acquire(
-                blocking=blocking,
-                blocking_timeout=blocking_timeout,
-            )
-        )
-
-    def release(self) -> None:
-        self._lock.release()
-
-    def owned(self) -> bool:
-        return bool(self._lock.owned())
-
-
-class RedisCacheBackend(CacheBackend):
-    """``CacheBackend`` implementation that delegates to a ``redis.Redis`` client.
-
-    This is a thin pass-through — every method maps 1-to-1 to the underlying
-    Redis command.  ``TenantRedis`` key-prefixing is handled by the client
-    itself (provided by ``get_redis_client``).
-    """
-
-    def __init__(self, redis_client: Redis) -> None:
-        self._r = redis_client
-
-    # -- basic key/value ---------------------------------------------------
-
-    def get(self, key: str) -> bytes | None:
-        val = self._r.get(key)
-        if val is None:
-            return None
-        if isinstance(val, bytes):
-            return val
-        return str(val).encode()
-
-    def set(
-        self,
-        key: str,
-        value: str | bytes | int | float,
-        ex: int | None = None,
-    ) -> None:
-        self._r.set(key, value, ex=ex)
-
-    def delete(self, key: str) -> None:
-        self._r.delete(key)
-
-    def exists(self, key: str) -> bool:
-        return bool(self._r.exists(key))
-
-    # -- TTL ---------------------------------------------------------------
-
-    def expire(self, key: str, seconds: int) -> None:
-        self._r.expire(key, seconds)
-
-    def ttl(self, key: str) -> int:
-        return cast(int, self._r.ttl(key))
-
-    # -- distributed lock --------------------------------------------------
-
-    def lock(self, name: str, timeout: float | None = None) -> CacheLock:
-        return RedisCacheLock(self._r.lock(name, timeout=timeout))
-
-    # -- blocking list (MCP OAuth BLPOP pattern) ---------------------------
-
-    def rpush(self, key: str, value: str | bytes) -> None:
-        self._r.rpush(key, value)
-
-    def blpop(self, keys: list[str], timeout: int = 0) -> tuple[bytes, bytes] | None:
-        result = cast(list[bytes] | None, self._r.blpop(keys, timeout=timeout))
-        if result is None:
-            return None
-        return (result[0], result[1])
--- a/backend/onyx/chat/chat_processing_checker.py
+++ b/backend/onyx/chat/chat_processing_checker.py
@@ -1,52 +1,57 @@
 from uuid import UUID

-from onyx.cache.interface import CacheBackend
+from redis.client import Redis

+# Redis key prefixes for chat message processing
 PREFIX = "chatprocessing"
 FENCE_PREFIX = f"{PREFIX}_fence"
 FENCE_TTL = 30 * 60  # 30 minutes


 def _get_fence_key(chat_session_id: UUID) -> str:
-    """Generate the cache key for a chat session processing fence.
+    """
+    Generate the Redis key for a chat session processing a message.

    Args:
        chat_session_id: The UUID of the chat session

    Returns:
-        The fence key string. Tenant isolation is handled automatically
-        by the cache backend (Redis key-prefixing or Postgres schema routing).
+        The fence key string (tenant_id is automatically added by the Redis client)
    """
    return f"{FENCE_PREFIX}_{chat_session_id}"


 def set_processing_status(
-    chat_session_id: UUID, cache: CacheBackend, value: bool
+    chat_session_id: UUID, redis_client: Redis, value: bool
 ) -> None:
-    """Set or clear the fence for a chat session processing a message.
+    """
+    Set or clear the fence for a chat session processing a message.

-    If the key exists, a message is being processed.
+    If the key exists, we are processing a message. If the key does not exist, we are not processing a message.

    Args:
        chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: The Redis client to use
        value: True to set the fence, False to clear it
    """
    fence_key = _get_fence_key(chat_session_id)
+
    if value:
-        cache.set(fence_key, 0, ex=FENCE_TTL)
+        redis_client.set(fence_key, 0, ex=FENCE_TTL)
    else:
-        cache.delete(fence_key)
+        redis_client.delete(fence_key)


-def is_chat_session_processing(chat_session_id: UUID, cache: CacheBackend) -> bool:
-    """Check if the chat session is processing a message.
+def is_chat_session_processing(chat_session_id: UUID, redis_client: Redis) -> bool:
+    """
+    Check if the chat session is processing a message.

    Args:
        chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: The Redis client to use

    Returns:
        True if the chat session is processing a message, False otherwise
    """
-    return cache.exists(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    return bool(redis_client.exists(fence_key))
--- a/backend/onyx/chat/llm_loop.py
+++ b/backend/onyx/chat/llm_loop.py
@@ -1,7 +1,6 @@
 import json
 import time
 from collections.abc import Callable
-from typing import Any
 from typing import Literal

 from sqlalchemy.orm import Session
@@ -36,6 +35,7 @@ from onyx.db.memory import add_memory
 from onyx.db.memory import update_memory_at_index
 from onyx.db.memory import UserMemoryContext
 from onyx.db.models import Persona
+from onyx.llm.constants import LlmProviderNames
 from onyx.llm.interfaces import LLM
 from onyx.llm.interfaces import LLMUserIdentity
 from onyx.llm.interfaces import ToolChoiceOptions
@@ -51,7 +51,6 @@ from onyx.tools.built_in_tools import STOPPING_TOOLS_NAMES
 from onyx.tools.interface import Tool
 from onyx.tools.models import ChatFile
 from onyx.tools.models import MemoryToolResponseSnapshot
-from onyx.tools.models import PythonToolRichResponse
 from onyx.tools.models import ToolCallInfo
 from onyx.tools.models import ToolCallKickoff
 from onyx.tools.models import ToolResponse
@@ -83,6 +82,28 @@ def _looks_like_xml_tool_call_payload(text: str | None) -> bool:
    )


+def _should_keep_bedrock_tool_definitions(
+    llm: object, simple_chat_history: list[ChatMessageSimple]
+) -> bool:
+    """Bedrock requires tool config when history includes toolUse/toolResult blocks."""
+    model_provider = getattr(getattr(llm, "config", None), "model_provider", None)
+    if model_provider not in {
+        LlmProviderNames.BEDROCK,
+        LlmProviderNames.BEDROCK_CONVERSE,
+    }:
+        return False
+
+    return any(
+        (
+            msg.message_type == MessageType.ASSISTANT
+            and msg.tool_calls
+            and len(msg.tool_calls) > 0
+        )
+        or msg.message_type == MessageType.TOOL_CALL_RESPONSE
+        for msg in simple_chat_history
+    )
+
+
 def _try_fallback_tool_extraction(
    llm_step_result: LlmStepResult,
    tool_choice: ToolChoiceOptions,
@@ -509,13 +530,11 @@ def _create_file_tool_metadata_message(
    """
    lines = [
        "You have access to the following files. Use the read_file tool to "
-        "read sections of any file. You MUST pass the file_id UUID (not the "
-        "filename) to read_file:"
+        "read sections of any file:"
    ]
    for meta in file_metadata:
        lines.append(
-            f'- file_id="{meta.file_id}" filename="{meta.filename}" '
-            f"(~{meta.approx_char_count:,} chars)"
+            f'- {meta.file_id}: "{meta.filename}" (~{meta.approx_char_count:,} chars)'
        )

    message_content = "\n".join(lines)
@@ -539,16 +558,12 @@ def _create_context_files_message(
    # Format as documents JSON as described in README
    documents_list = []
    for idx, file_text in enumerate(context_files.file_texts, start=1):
-        title = (
-            context_files.file_metadata[idx - 1].filename
-            if idx - 1 < len(context_files.file_metadata)
-            else None
+        documents_list.append(
+            {
+                "document": idx,
+                "contents": file_text,
+            }
        )
-        entry: dict[str, Any] = {"document": idx}
-        if title:
-            entry["title"] = title
-        entry["contents"] = file_text
-        documents_list.append(entry)

    documents_json = json.dumps({"documents": documents_list}, indent=2)
    message_content = f"Here are some documents provided for context, they may not all be relevant:\n{documents_json}"
@@ -663,7 +678,12 @@ def run_llm_loop(
            elif out_of_cycles or ran_image_gen:
                # Last cycle, no tools allowed, just answer!
                tool_choice = ToolChoiceOptions.NONE
-                final_tools = []
+                # Bedrock requires tool config in requests that include toolUse/toolResult history.
+                final_tools = (
+                    tools
+                    if _should_keep_bedrock_tool_definitions(llm, simple_chat_history)
+                    else []
+                )
            else:
                tool_choice = ToolChoiceOptions.AUTO
                final_tools = tools
@@ -939,13 +959,6 @@ def run_llm_loop(
                ):
                    generated_images = tool_response.rich_response.generated_images

-                # Extract generated_files if this is a code interpreter response
-                generated_files = None
-                if isinstance(tool_response.rich_response, PythonToolRichResponse):
-                    generated_files = (
-                        tool_response.rich_response.generated_files or None
-                    )
-
                # Persist memory if this is a memory tool response
                memory_snapshot: MemoryToolResponseSnapshot | None = None
                if isinstance(tool_response.rich_response, MemoryToolResponse):
@@ -997,7 +1010,6 @@ def run_llm_loop(
                    tool_call_response=saved_response,
                    search_docs=displayed_docs or search_docs,
                    generated_images=generated_images,
-                    generated_files=generated_files,
                )
                # Add to state container for partial save support
                state_container.add_tool_call(tool_call_info)
--- a/backend/onyx/chat/llm_step.py
+++ b/backend/onyx/chat/llm_step.py
@@ -15,7 +15,6 @@ from onyx.chat.citation_processor import DynamicCitationProcessor
 from onyx.chat.emitter import Emitter
 from onyx.chat.models import ChatMessageSimple
 from onyx.chat.models import LlmStepResult
-from onyx.chat.tool_call_args_streaming import maybe_emit_argument_delta
 from onyx.configs.app_configs import LOG_ONYX_MODEL_INTERACTIONS
 from onyx.configs.app_configs import PROMPT_CACHE_CHAT_HISTORY
 from onyx.configs.constants import MessageType
@@ -55,9 +54,7 @@ from onyx.server.query_and_chat.streaming_models import ReasoningStart
 from onyx.tools.models import ToolCallKickoff
 from onyx.tracing.framework.create import generation_span
 from onyx.utils.b64 import get_image_type_from_bytes
-from onyx.utils.jsonriver import Parser
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string
 from onyx.utils.text_processing import find_all_json_objects

 logger = setup_logger()
@@ -169,6 +166,15 @@ def _find_function_calls_open_marker(text_lower: str) -> int:
        search_from = idx + 1


+def _sanitize_llm_output(value: str) -> str:
+    """Remove characters that PostgreSQL's text/JSONB types cannot store.
+
+    - NULL bytes (\x00): Not allowed in PostgreSQL text types
+    - UTF-16 surrogates (\ud800-\udfff): Invalid in UTF-8 encoding
+    """
+    return "".join(c for c in value if c != "\x00" and not ("\ud800" <= c <= "\udfff"))
+
+
 def _try_parse_json_string(value: Any) -> Any:
    """Attempt to parse a JSON string value into its Python equivalent.

@@ -216,7 +222,9 @@ def _parse_tool_args_to_dict(raw_args: Any) -> dict[str, Any]:
    if isinstance(raw_args, dict):
        # Parse any string values that look like JSON arrays/objects
        return {
-            k: _try_parse_json_string(sanitize_string(v) if isinstance(v, str) else v)
+            k: _try_parse_json_string(
+                _sanitize_llm_output(v) if isinstance(v, str) else v
+            )
            for k, v in raw_args.items()
        }

@@ -224,7 +232,7 @@ def _parse_tool_args_to_dict(raw_args: Any) -> dict[str, Any]:
        return {}

    # Sanitize before parsing to remove NULL bytes and surrogates
-    raw_args = sanitize_string(raw_args)
+    raw_args = _sanitize_llm_output(raw_args)

    try:
        parsed1: Any = json.loads(raw_args)
@@ -537,12 +545,12 @@ def _extract_xml_attribute(attrs: str, attr_name: str) -> str | None:
    )
    if not attr_match:
        return None
-    return sanitize_string(unescape(attr_match.group(2).strip()))
+    return _sanitize_llm_output(unescape(attr_match.group(2).strip()))


 def _parse_xml_parameter_value(raw_value: str, string_attr: str | None) -> Any:
    """Parse a parameter value from XML-style tool call payloads."""
-    value = sanitize_string(unescape(raw_value).strip())
+    value = _sanitize_llm_output(unescape(raw_value).strip())

    if string_attr and string_attr.lower() == "true":
        return value
@@ -561,7 +569,6 @@ def _resolve_tool_arguments(obj: dict[str, Any]) -> dict[str, Any] | None:
    """
    arguments = obj.get("arguments", obj.get("parameters", {}))
    if isinstance(arguments, str):
-        arguments = sanitize_string(arguments)
        try:
            arguments = json.loads(arguments)
        except json.JSONDecodeError:
@@ -1011,7 +1018,6 @@ def run_llm_step_pkt_generator(
        )

    id_to_tool_call_map: dict[int, dict[str, Any]] = {}
-    arg_parsers: dict[int, Parser] = {}
    reasoning_start = False
    answer_start = False
    accumulated_reasoning = ""
@@ -1218,14 +1224,7 @@ def run_llm_step_pkt_generator(
                yield from _close_reasoning_if_active()

                for tool_call_delta in delta.tool_calls:
-                    # maybe_emit depends and update being called first and attaching the delta
                    _update_tool_call_with_delta(id_to_tool_call_map, tool_call_delta)
-                    yield from maybe_emit_argument_delta(
-                        tool_calls_in_progress=id_to_tool_call_map,
-                        tool_call_delta=tool_call_delta,
-                        placement=_current_placement(),
-                        parsers=arg_parsers,
-                    )

        # Flush any tail text buffered while checking for split "<function_calls" markers.
        filtered_content_tail = xml_tool_call_content_filter.flush()
--- a/backend/onyx/chat/process_message.py
+++ b/backend/onyx/chat/process_message.py
@@ -11,10 +11,9 @@ from contextvars import Token
 from uuid import UUID

 from pydantic import BaseModel
+from redis.client import Redis
 from sqlalchemy.orm import Session

-from onyx.cache.factory import get_cache_backend
-from onyx.cache.interface import CacheBackend
 from onyx.chat.chat_processing_checker import set_processing_status
 from onyx.chat.chat_state import ChatStateContainer
 from onyx.chat.chat_state import run_chat_loop_with_state_containers
@@ -80,6 +79,7 @@ from onyx.llm.request_context import reset_llm_mock_response
 from onyx.llm.request_context import set_llm_mock_response
 from onyx.llm.utils import litellm_exception_to_error_msg
 from onyx.onyxbot.slack.models import SlackContext
+from onyx.redis.redis_pool import get_redis_client
 from onyx.server.query_and_chat.models import AUTO_PLACE_AFTER_LATEST_MESSAGE
 from onyx.server.query_and_chat.models import MessageResponseIDInfo
 from onyx.server.query_and_chat.models import SendMessageRequest
@@ -448,7 +448,7 @@ def handle_stream_message_objects(

    llm: LLM | None = None
    chat_session: ChatSession | None = None
-    cache: CacheBackend | None = None
+    redis_client: Redis | None = None

    user_id = user.id
    if user.is_anonymous:
@@ -809,19 +809,19 @@ def handle_stream_message_objects(
            )
            simple_chat_history.insert(0, summary_simple)

-        cache = get_cache_backend()
+        redis_client = get_redis_client()

        reset_cancel_status(
            chat_session.id,
-            cache,
+            redis_client,
        )

        def check_is_connected() -> bool:
-            return check_stop_signal(chat_session.id, cache)
+            return check_stop_signal(chat_session.id, redis_client)

        set_processing_status(
            chat_session_id=chat_session.id,
-            cache=cache,
+            redis_client=redis_client,
            value=True,
        )

@@ -968,10 +968,10 @@ def handle_stream_message_objects(
            reset_llm_mock_response(mock_response_token)

        try:
-            if cache is not None and chat_session is not None:
+            if redis_client is not None and chat_session is not None:
                set_processing_status(
                    chat_session_id=chat_session.id,
-                    cache=cache,
+                    redis_client=redis_client,
                    value=False,
                )
        except Exception:
--- a/backend/onyx/chat/save_chat.py
+++ b/backend/onyx/chat/save_chat.py
@@ -1,5 +1,4 @@
 import json
-import mimetypes

 from sqlalchemy.orm import Session

@@ -13,42 +12,14 @@ from onyx.db.chat import create_db_search_doc
 from onyx.db.models import ChatMessage
 from onyx.db.models import ToolCall
 from onyx.db.tools import create_tool_call_no_commit
-from onyx.file_store.models import FileDescriptor
 from onyx.natural_language_processing.utils import BaseTokenizer
 from onyx.natural_language_processing.utils import get_tokenizer
-from onyx.server.query_and_chat.chat_utils import mime_type_to_chat_file_type
 from onyx.tools.models import ToolCallInfo
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string

 logger = setup_logger()


-def _extract_referenced_file_descriptors(
-    tool_calls: list[ToolCallInfo],
-    message_text: str,
-) -> list[FileDescriptor]:
-    """Extract FileDescriptors for code interpreter files referenced in the message text."""
-    descriptors: list[FileDescriptor] = []
-    for tool_call_info in tool_calls:
-        if not tool_call_info.generated_files:
-            continue
-        for gen_file in tool_call_info.generated_files:
-            file_id = (
-                gen_file.file_link.rsplit("/", 1)[-1] if gen_file.file_link else ""
-            )
-            if file_id and file_id in message_text:
-                mime_type, _ = mimetypes.guess_type(gen_file.filename)
-                descriptors.append(
-                    FileDescriptor(
-                        id=file_id,
-                        type=mime_type_to_chat_file_type(mime_type),
-                        name=gen_file.filename,
-                    )
-                )
-    return descriptors
-
-
 def _create_and_link_tool_calls(
    tool_calls: list[ToolCallInfo],
    assistant_message: ChatMessage,
@@ -202,13 +173,8 @@ def save_chat_turn(
        pre_answer_processing_time: Duration of processing before answer starts (in seconds)
    """
    # 1. Update ChatMessage with message content, reasoning tokens, and token count
-    sanitized_message_text = (
-        sanitize_string(message_text) if message_text else message_text
-    )
-    assistant_message.message = sanitized_message_text
-    assistant_message.reasoning_tokens = (
-        sanitize_string(reasoning_tokens) if reasoning_tokens else reasoning_tokens
-    )
+    assistant_message.message = message_text
+    assistant_message.reasoning_tokens = reasoning_tokens
    assistant_message.is_clarification = is_clarification

    # Use pre-answer processing time (captured when MESSAGE_START was emitted)
@@ -218,10 +184,8 @@ def save_chat_turn(
    # Calculate token count using default tokenizer, when storing, this should not use the LLM
    # specific one so we use a system default tokenizer here.
    default_tokenizer = get_tokenizer(None, None)
-    if sanitized_message_text:
-        assistant_message.token_count = len(
-            default_tokenizer.encode(sanitized_message_text)
-        )
+    if message_text:
+        assistant_message.token_count = len(default_tokenizer.encode(message_text))
    else:
        assistant_message.token_count = 0

@@ -333,16 +297,5 @@ def save_chat_turn(
        citation_number_to_search_doc_id if citation_number_to_search_doc_id else None
    )

-    # 8. Attach code interpreter generated files that the assistant actually
-    # referenced in its response, so they are available via load_all_chat_files
-    # on subsequent turns. Files not mentioned are intermediate artifacts.
-    if sanitized_message_text:
-        referenced = _extract_referenced_file_descriptors(
-            tool_calls, sanitized_message_text
-        )
-        if referenced:
-            existing_files = assistant_message.files or []
-            assistant_message.files = existing_files + referenced
-
    # Finally save the messages, tool calls, and docs
    db_session.commit()
--- a/backend/onyx/chat/stop_signal_checker.py
+++ b/backend/onyx/chat/stop_signal_checker.py
@@ -1,58 +1,65 @@
 from uuid import UUID

-from onyx.cache.interface import CacheBackend
+from redis.client import Redis

+# Redis key prefixes for chat session stop signals
 PREFIX = "chatsessionstop"
 FENCE_PREFIX = f"{PREFIX}_fence"
-FENCE_TTL = 10 * 60  # 10 minutes
+FENCE_TTL = 10 * 60  # 10 minutes - defensive TTL to prevent memory leaks


 def _get_fence_key(chat_session_id: UUID) -> str:
-    """Generate the cache key for a chat session stop signal fence.
+    """
+    Generate the Redis key for a chat session stop signal fence.

    Args:
        chat_session_id: The UUID of the chat session

    Returns:
-        The fence key string. Tenant isolation is handled automatically
-        by the cache backend (Redis key-prefixing or Postgres schema routing).
+        The fence key string (tenant_id is automatically added by the Redis client)
    """
    return f"{FENCE_PREFIX}_{chat_session_id}"


-def set_fence(chat_session_id: UUID, cache: CacheBackend, value: bool) -> None:
-    """Set or clear the stop signal fence for a chat session.
+def set_fence(chat_session_id: UUID, redis_client: Redis, value: bool) -> None:
+    """
+    Set or clear the stop signal fence for a chat session.

    Args:
        chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use (tenant-aware client that auto-prefixes keys)
        value: True to set the fence (stop signal), False to clear it
    """
    fence_key = _get_fence_key(chat_session_id)
    if not value:
-        cache.delete(fence_key)
+        redis_client.delete(fence_key)
        return
-    cache.set(fence_key, 0, ex=FENCE_TTL)
+
+    redis_client.set(fence_key, 0, ex=FENCE_TTL)


-def is_connected(chat_session_id: UUID, cache: CacheBackend) -> bool:
-    """Check if the chat session should continue (not stopped).
+def is_connected(chat_session_id: UUID, redis_client: Redis) -> bool:
+    """
+    Check if the chat session should continue (not stopped).

    Args:
        chat_session_id: The UUID of the chat session to check
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use for checking the stop signal (tenant-aware client that auto-prefixes keys)

    Returns:
        True if the session should continue, False if it should stop
    """
-    return not cache.exists(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    return not bool(redis_client.exists(fence_key))


-def reset_cancel_status(chat_session_id: UUID, cache: CacheBackend) -> None:
-    """Clear the stop signal for a chat session.
+def reset_cancel_status(chat_session_id: UUID, redis_client: Redis) -> None:
+    """
+    Clear the stop signal for a chat session.

    Args:
        chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use (tenant-aware client that auto-prefixes keys)
    """
-    cache.delete(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    redis_client.delete(fence_key)
--- a/backend/onyx/chat/tool_call_args_streaming.py
+++ b/backend/onyx/chat/tool_call_args_streaming.py
@@ -1,77 +0,0 @@
-from collections.abc import Generator
-from collections.abc import Mapping
-from typing import Any
-from typing import Type
-
-from onyx.llm.model_response import ChatCompletionDeltaToolCall
-from onyx.server.query_and_chat.placement import Placement
-from onyx.server.query_and_chat.streaming_models import Packet
-from onyx.server.query_and_chat.streaming_models import ToolCallArgumentDelta
-from onyx.tools.built_in_tools import TOOL_NAME_TO_CLASS
-from onyx.tools.interface import Tool
-from onyx.utils.jsonriver import Parser
-
-
-def _get_tool_class(
-    tool_calls_in_progress: Mapping[int, Mapping[str, Any]],
-    tool_call_delta: ChatCompletionDeltaToolCall,
-) -> Type[Tool] | None:
-    """Look up the Tool subclass for a streaming tool call delta."""
-    tool_name = tool_calls_in_progress.get(tool_call_delta.index, {}).get("name")
-    if not tool_name:
-        return None
-    return TOOL_NAME_TO_CLASS.get(tool_name)
-
-
-def maybe_emit_argument_delta(
-    tool_calls_in_progress: Mapping[int, Mapping[str, Any]],
-    tool_call_delta: ChatCompletionDeltaToolCall,
-    placement: Placement,
-    parsers: dict[int, Parser],
-) -> Generator[Packet, None, None]:
-    """Emit decoded tool-call argument deltas to the frontend.
-
-    Uses a ``jsonriver.Parser`` per tool-call index to incrementally parse
-    the JSON argument string and extract only the newly-appended content
-    for each string-valued argument.
-
-    NOTE: Non-string arguments (numbers, booleans, null, arrays, objects)
-    are skipped — they are available in the final tool-call kickoff packet.
-
-    ``parsers`` is a mutable dict keyed by tool-call index. A new
-    ``Parser`` is created automatically for each new index.
-    """
-    tool_cls = _get_tool_class(tool_calls_in_progress, tool_call_delta)
-    if not tool_cls or not tool_cls.should_emit_argument_deltas():
-        return
-
-    fn = tool_call_delta.function
-    delta_fragment = fn.arguments if fn else None
-    if not delta_fragment:
-        return
-
-    idx = tool_call_delta.index
-    if idx not in parsers:
-        parsers[idx] = Parser()
-    parser = parsers[idx]
-
-    deltas = parser.feed(delta_fragment)
-
-    argument_deltas: dict[str, str] = {}
-    for delta in deltas:
-        if isinstance(delta, dict):
-            for key, value in delta.items():
-                if isinstance(value, str):
-                    argument_deltas[key] = argument_deltas.get(key, "") + value
-
-    if not argument_deltas:
-        return
-
-    tc_data = tool_calls_in_progress[tool_call_delta.index]
-    yield Packet(
-        placement=placement,
-        obj=ToolCallArgumentDelta(
-            tool_type=tc_data.get("name", ""),
-            argument_deltas=argument_deltas,
-        ),
-    )
--- a/backend/onyx/configs/app_configs.py
+++ b/backend/onyx/configs/app_configs.py
@@ -6,7 +6,6 @@ from datetime import timezone
 from typing import cast

 from onyx.auth.schemas import AuthBackend
-from onyx.cache.interface import CacheBackendType
 from onyx.configs.constants import AuthType
 from onyx.configs.constants import QueryHistoryType
 from onyx.file_processing.enums import HtmlBasedConnectorTransformLinksStrategy
@@ -55,12 +54,6 @@ DISABLE_USER_KNOWLEDGE = os.environ.get("DISABLE_USER_KNOWLEDGE", "").lower() ==
 # are disabled but core chat, tools, user file uploads, and Projects still work.
 DISABLE_VECTOR_DB = os.environ.get("DISABLE_VECTOR_DB", "").lower() == "true"

-# Which backend to use for caching, locks, and ephemeral state.
-# "redis" (default) or "postgres" (only valid when DISABLE_VECTOR_DB=true).
-CACHE_BACKEND = CacheBackendType(
-    os.environ.get("CACHE_BACKEND", CacheBackendType.REDIS)
-)
-
 # Maximum token count for a single uploaded file. Files exceeding this are rejected.
 # Defaults to 100k tokens (or 10M when vector DB is disabled).
 _DEFAULT_FILE_TOKEN_LIMIT = 10_000_000 if DISABLE_VECTOR_DB else 100_000
@@ -68,10 +61,6 @@ FILE_TOKEN_COUNT_THRESHOLD = int(
    os.environ.get("FILE_TOKEN_COUNT_THRESHOLD", str(_DEFAULT_FILE_TOKEN_LIMIT))
 )

-# Maximum upload size for a single user file (chat/projects) in MB.
-USER_FILE_MAX_UPLOAD_SIZE_MB = int(os.environ.get("USER_FILE_MAX_UPLOAD_SIZE_MB") or 50)
-USER_FILE_MAX_UPLOAD_SIZE_BYTES = USER_FILE_MAX_UPLOAD_SIZE_MB * 1024 * 1024
-
 # If set to true, will show extra/uncommon connectors in the "Other" category
 SHOW_EXTRA_CONNECTORS = os.environ.get("SHOW_EXTRA_CONNECTORS", "").lower() == "true"

@@ -499,7 +488,14 @@ CELERY_WORKER_PRIMARY_POOL_OVERFLOW = int(
    os.environ.get("CELERY_WORKER_PRIMARY_POOL_OVERFLOW") or 4
 )

-# Individual worker concurrency settings
+# Consolidated background worker (light, docprocessing, docfetching, heavy, monitoring, user_file_processing)
+# separate workers' defaults: light=24, docprocessing=6, docfetching=1, heavy=4, kg=2, monitoring=1, user_file=2
+# Total would be 40, but we use a more conservative default of 20 for the consolidated worker
+CELERY_WORKER_BACKGROUND_CONCURRENCY = int(
+    os.environ.get("CELERY_WORKER_BACKGROUND_CONCURRENCY") or 20
+)
+
+# Individual worker concurrency settings (used when USE_LIGHTWEIGHT_BACKGROUND_WORKER is False or on Kuberenetes deployments)
 CELERY_WORKER_HEAVY_CONCURRENCY = int(
    os.environ.get("CELERY_WORKER_HEAVY_CONCURRENCY") or 4
 )
@@ -816,9 +812,7 @@ RERANK_COUNT = int(os.environ.get("RERANK_COUNT") or 1000)
 # Tool Configs
 #####
 # Code Interpreter Service Configuration
-CODE_INTERPRETER_BASE_URL = os.environ.get(
-    "CODE_INTERPRETER_BASE_URL", "http://localhost:8000"
-)
+CODE_INTERPRETER_BASE_URL = os.environ.get("CODE_INTERPRETER_BASE_URL")

 CODE_INTERPRETER_DEFAULT_TIMEOUT_MS = int(
    os.environ.get("CODE_INTERPRETER_DEFAULT_TIMEOUT_MS") or 60_000
@@ -899,9 +893,6 @@ CUSTOM_ANSWER_VALIDITY_CONDITIONS = json.loads(
 )

 VESPA_REQUEST_TIMEOUT = int(os.environ.get("VESPA_REQUEST_TIMEOUT") or "15")
-VESPA_MIGRATION_REQUEST_TIMEOUT_S = int(
-    os.environ.get("VESPA_MIGRATION_REQUEST_TIMEOUT_S") or "120"
-)

 SYSTEM_RECURSION_LIMIT = int(os.environ.get("SYSTEM_RECURSION_LIMIT") or "1000")

--- a/backend/onyx/configs/constants.py
+++ b/backend/onyx/configs/constants.py
@@ -84,6 +84,7 @@ POSTGRES_CELERY_WORKER_LIGHT_APP_NAME = "celery_worker_light"
 POSTGRES_CELERY_WORKER_DOCPROCESSING_APP_NAME = "celery_worker_docprocessing"
 POSTGRES_CELERY_WORKER_DOCFETCHING_APP_NAME = "celery_worker_docfetching"
 POSTGRES_CELERY_WORKER_INDEXING_CHILD_APP_NAME = "celery_worker_indexing_child"
+POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME = "celery_worker_background"
 POSTGRES_CELERY_WORKER_HEAVY_APP_NAME = "celery_worker_heavy"
 POSTGRES_CELERY_WORKER_MONITORING_APP_NAME = "celery_worker_monitoring"
 POSTGRES_CELERY_WORKER_USER_FILE_PROCESSING_APP_NAME = (
--- a/backend/onyx/connectors/confluence/connector.py
+++ b/backend/onyx/connectors/confluence/connector.py
@@ -943,9 +943,6 @@ class ConfluenceConnector(
                        if include_permissions
                        else None
                    ),
-                    parent_hierarchy_raw_node_id=self._get_parent_hierarchy_raw_id(
-                        page
-                    ),
                )
            )

@@ -995,7 +992,6 @@ class ConfluenceConnector(
                            if include_permissions
                            else None
                        ),
-                        parent_hierarchy_raw_node_id=page_id,
                    )
                )

--- a/backend/onyx/connectors/google_drive/connector.py
+++ b/backend/onyx/connectors/google_drive/connector.py
@@ -1722,7 +1722,6 @@ class GoogleDriveConnector(
                        primary_admin_email=self.primary_admin_email,
                        google_domain=self.google_domain,
                    ),
-                    retriever_email=file.user_email,
                ):
                    slim_batch.append(doc)

--- a/backend/onyx/connectors/google_drive/doc_conversion.py
+++ b/backend/onyx/connectors/google_drive/doc_conversion.py
@@ -476,7 +476,6 @@ def _get_external_access_for_raw_gdrive_file(
    company_domain: str,
    retriever_drive_service: GoogleDriveService | None,
    admin_drive_service: GoogleDriveService,
-    fallback_user_email: str,
    add_prefix: bool = False,
 ) -> ExternalAccess:
    """
@@ -485,8 +484,6 @@ def _get_external_access_for_raw_gdrive_file(
    add_prefix: When True, prefix group IDs with source type (for indexing path).
               When False (default), leave unprefixed (for permission sync path
               where upsert_document_external_perms handles prefixing).
-    fallback_user_email: When permission info can't be retrieved (e.g. externally-owned
-               files), fall back to granting access to this user.
    """
    external_access_fn = cast(
        Callable[
@@ -495,7 +492,6 @@ def _get_external_access_for_raw_gdrive_file(
                str,
                GoogleDriveService | None,
                GoogleDriveService,
-                str,
                bool,
            ],
            ExternalAccess,
@@ -511,7 +507,6 @@ def _get_external_access_for_raw_gdrive_file(
        company_domain,
        retriever_drive_service,
        admin_drive_service,
-        fallback_user_email,
        add_prefix,
    )

@@ -677,7 +672,6 @@ def _convert_drive_item_to_document(
                    creds, user_email=permission_sync_context.primary_admin_email
                ),
                add_prefix=True,  # Indexing path - prefix here
-                fallback_user_email=retriever_email,
            )
            if permission_sync_context
            else None
@@ -759,7 +753,6 @@ def build_slim_document(
    # if not specified, we will not sync permissions
    # will also be a no-op if EE is not enabled
    permission_sync_context: PermissionSyncContext | None,
-    retriever_email: str,
 ) -> SlimDocument | None:
    if file.get("mimeType") in [DRIVE_FOLDER_TYPE, DRIVE_SHORTCUT_TYPE]:
        return None
@@ -781,7 +774,6 @@ def build_slim_document(
                creds,
                user_email=permission_sync_context.primary_admin_email,
            ),
-            fallback_user_email=retriever_email,
        )
        if permission_sync_context
        else None
@@ -789,5 +781,4 @@ def build_slim_document(
    return SlimDocument(
        id=onyx_document_id_from_drive_file(file),
        external_access=external_access,
-        parent_hierarchy_raw_node_id=(file.get("parents") or [None])[0],
    )
--- a/backend/onyx/connectors/jira/connector.py
+++ b/backend/onyx/connectors/jira/connector.py
@@ -902,11 +902,6 @@ class JiraConnector(
                        external_access=self._get_project_permissions(
                            project_key, add_prefix=False
                        ),
-                        parent_hierarchy_raw_node_id=(
-                            self._get_parent_hierarchy_raw_node_id(issue, project_key)
-                            if project_key
-                            else None
-                        ),
                    )
                )
                current_offset += 1
--- a/backend/onyx/connectors/models.py
+++ b/backend/onyx/connectors/models.py
@@ -385,7 +385,6 @@ class IndexingDocument(Document):
 class SlimDocument(BaseModel):
    id: str
    external_access: ExternalAccess | None = None
-    parent_hierarchy_raw_node_id: str | None = None


 class HierarchyNode(BaseModel):
--- a/backend/onyx/connectors/sharepoint/connector.py
+++ b/backend/onyx/connectors/sharepoint/connector.py
@@ -772,7 +772,6 @@ def _convert_driveitem_to_slim_document(
    drive_name: str,
    ctx: ClientContext,
    graph_client: GraphClient,
-    parent_hierarchy_raw_node_id: str | None = None,
 ) -> SlimDocument:
    if driveitem.id is None:
        raise ValueError("DriveItem ID is required")
@@ -788,15 +787,11 @@ def _convert_driveitem_to_slim_document(
    return SlimDocument(
        id=driveitem.id,
        external_access=external_access,
-        parent_hierarchy_raw_node_id=parent_hierarchy_raw_node_id,
    )


 def _convert_sitepage_to_slim_document(
-    site_page: dict[str, Any],
-    ctx: ClientContext | None,
-    graph_client: GraphClient,
-    parent_hierarchy_raw_node_id: str | None = None,
+    site_page: dict[str, Any], ctx: ClientContext | None, graph_client: GraphClient
 ) -> SlimDocument:
    """Convert a SharePoint site page to a SlimDocument object."""
    if site_page.get("id") is None:
@@ -813,7 +808,6 @@ def _convert_sitepage_to_slim_document(
    return SlimDocument(
        id=id,
        external_access=external_access,
-        parent_hierarchy_raw_node_id=parent_hierarchy_raw_node_id,
    )


@@ -1600,22 +1594,12 @@ class SharepointConnector(
                            )
                        )

-                    parent_hierarchy_url: str | None = None
-                    if drive_web_url:
-                        parent_hierarchy_url = self._get_parent_hierarchy_url(
-                            site_url, drive_web_url, drive_name, driveitem
-                        )
-
                    try:
                        logger.debug(f"Processing: {driveitem.web_url}")
                        ctx = self._create_rest_client_context(site_descriptor.url)
                        doc_batch.append(
                            _convert_driveitem_to_slim_document(
-                                driveitem,
-                                drive_name,
-                                ctx,
-                                self.graph_client,
-                                parent_hierarchy_raw_node_id=parent_hierarchy_url,
+                                driveitem, drive_name, ctx, self.graph_client
                            )
                        )
                    except Exception as e:
@@ -1635,10 +1619,7 @@ class SharepointConnector(
                    ctx = self._create_rest_client_context(site_descriptor.url)
                    doc_batch.append(
                        _convert_sitepage_to_slim_document(
-                            site_page,
-                            ctx,
-                            self.graph_client,
-                            parent_hierarchy_raw_node_id=site_descriptor.url,
+                            site_page, ctx, self.graph_client
                        )
                    )
                    if len(doc_batch) >= SLIM_BATCH_SIZE:
--- a/backend/onyx/connectors/slack/connector.py
+++ b/backend/onyx/connectors/slack/connector.py
@@ -565,7 +565,6 @@ def _get_all_doc_ids(
                            channel_id=channel_id, thread_ts=message["ts"]
                        ),
                        external_access=external_access,
-                        parent_hierarchy_raw_node_id=channel_id,
                    )
                )

--- a/backend/onyx/db/chat.py
+++ b/backend/onyx/db/chat.py
@@ -38,7 +38,6 @@ from onyx.llm.override_models import LLMOverride
 from onyx.llm.override_models import PromptOverride
 from onyx.server.query_and_chat.models import ChatMessageDetail
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string


 logger = setup_logger()
@@ -99,7 +98,6 @@ def get_chat_sessions_by_user(
    db_session: Session,
    include_onyxbot_flows: bool = False,
    limit: int = 50,
-    before: datetime | None = None,
    project_id: int | None = None,
    only_non_project_chats: bool = False,
    include_failed_chats: bool = False,
@@ -114,9 +112,6 @@ def get_chat_sessions_by_user(
    if deleted is not None:
        stmt = stmt.where(ChatSession.deleted == deleted)

-    if before is not None:
-        stmt = stmt.where(ChatSession.time_updated < before)
-
    if limit:
        stmt = stmt.limit(limit)

@@ -676,43 +671,58 @@ def set_as_latest_chat_message(
    db_session.commit()


+def _sanitize_for_postgres(value: str) -> str:
+    """Remove NUL (0x00) characters from strings as PostgreSQL doesn't allow them."""
+    sanitized = value.replace("\x00", "")
+    if value and not sanitized:
+        logger.warning("Sanitization removed all characters from string")
+    return sanitized
+
+
+def _sanitize_list_for_postgres(values: list[str]) -> list[str]:
+    """Remove NUL (0x00) characters from all strings in a list."""
+    return [_sanitize_for_postgres(v) for v in values]
+
+
 def create_db_search_doc(
    server_search_doc: ServerSearchDoc,
    db_session: Session,
    commit: bool = True,
 ) -> DBSearchDoc:
+    # Sanitize string fields to remove NUL characters (PostgreSQL doesn't allow them)
    db_search_doc = DBSearchDoc(
-        document_id=sanitize_string(server_search_doc.document_id),
+        document_id=_sanitize_for_postgres(server_search_doc.document_id),
        chunk_ind=server_search_doc.chunk_ind,
-        semantic_id=sanitize_string(server_search_doc.semantic_identifier),
+        semantic_id=_sanitize_for_postgres(server_search_doc.semantic_identifier),
        link=(
-            sanitize_string(server_search_doc.link)
+            _sanitize_for_postgres(server_search_doc.link)
            if server_search_doc.link is not None
            else None
        ),
-        blurb=sanitize_string(server_search_doc.blurb),
+        blurb=_sanitize_for_postgres(server_search_doc.blurb),
        source_type=server_search_doc.source_type,
        boost=server_search_doc.boost,
        hidden=server_search_doc.hidden,
        doc_metadata=server_search_doc.metadata,
        is_relevant=server_search_doc.is_relevant,
        relevance_explanation=(
-            sanitize_string(server_search_doc.relevance_explanation)
+            _sanitize_for_postgres(server_search_doc.relevance_explanation)
            if server_search_doc.relevance_explanation is not None
            else None
        ),
+        # For docs further down that aren't reranked, we can't use the retrieval score
        score=server_search_doc.score or 0.0,
-        match_highlights=[
-            sanitize_string(h) for h in server_search_doc.match_highlights
-        ],
+        match_highlights=_sanitize_list_for_postgres(
+            server_search_doc.match_highlights
+        ),
        updated_at=server_search_doc.updated_at,
        primary_owners=(
-            [sanitize_string(o) for o in server_search_doc.primary_owners]
+            _sanitize_list_for_postgres(server_search_doc.primary_owners)
            if server_search_doc.primary_owners is not None
            else None
        ),
        secondary_owners=(
-            [sanitize_string(o) for o in server_search_doc.secondary_owners]
+            _sanitize_list_for_postgres(server_search_doc.secondary_owners)
            if server_search_doc.secondary_owners is not None
            else None
        ),
--- a/backend/onyx/db/document_set.py
+++ b/backend/onyx/db/document_set.py
@@ -13,7 +13,6 @@ from sqlalchemy.orm import aliased
 from sqlalchemy.orm import selectinload
 from sqlalchemy.orm import Session

-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.db.connector_credential_pair import get_cc_pair_groups_for_ids
 from onyx.db.connector_credential_pair import get_connector_credential_pairs
 from onyx.db.enums import AccessType
@@ -247,7 +246,6 @@ def insert_document_set(
            description=document_set_creation_request.description,
            user_id=user_id,
            is_public=document_set_creation_request.is_public,
-            is_up_to_date=DISABLE_VECTOR_DB,
            time_last_modified_by_user=func.now(),
        )
        db_session.add(new_document_set_row)
@@ -338,8 +336,7 @@ def update_document_set(
            )

        document_set_row.description = document_set_update_request.description
-        if not DISABLE_VECTOR_DB:
-            document_set_row.is_up_to_date = False
+        document_set_row.is_up_to_date = False
        document_set_row.is_public = document_set_update_request.is_public
        document_set_row.time_last_modified_by_user = func.now()
        versioned_private_doc_set_fn = fetch_versioned_implementation(
--- a/backend/onyx/db/enums.py
+++ b/backend/onyx/db/enums.py
@@ -186,7 +186,6 @@ class EmbeddingPrecision(str, PyEnum):

 class UserFileStatus(str, PyEnum):
    PROCESSING = "PROCESSING"
-    INDEXING = "INDEXING"
    COMPLETED = "COMPLETED"
    FAILED = "FAILED"
    CANCELED = "CANCELED"
--- a/backend/onyx/db/hierarchy.py
+++ b/backend/onyx/db/hierarchy.py
@@ -1,7 +1,5 @@
 """CRUD operations for HierarchyNode."""

-from collections import defaultdict
-
 from sqlalchemy import select
 from sqlalchemy.orm import Session

@@ -527,53 +525,6 @@ def get_document_parent_hierarchy_node_ids(
    return {doc_id: parent_id for doc_id, parent_id in results}


-def update_document_parent_hierarchy_nodes(
-    db_session: Session,
-    doc_parent_map: dict[str, int | None],
-    commit: bool = True,
-) -> int:
-    """Bulk-update Document.parent_hierarchy_node_id for multiple documents.
-
-    Only updates rows whose current value differs from the desired value to
-    avoid unnecessary writes.
-
-    Args:
-        db_session: SQLAlchemy session
-        doc_parent_map: Mapping of document_id → desired parent_hierarchy_node_id
-        commit: Whether to commit the transaction
-
-    Returns:
-        Number of documents actually updated
-    """
-    if not doc_parent_map:
-        return 0
-
-    doc_ids = list(doc_parent_map.keys())
-    existing = get_document_parent_hierarchy_node_ids(db_session, doc_ids)
-
-    by_parent: dict[int | None, list[str]] = defaultdict(list)
-    for doc_id, desired_parent_id in doc_parent_map.items():
-        current = existing.get(doc_id)
-        if current == desired_parent_id or doc_id not in existing:
-            continue
-        by_parent[desired_parent_id].append(doc_id)
-
-    updated = 0
-    for desired_parent_id, ids in by_parent.items():
-        db_session.query(Document).filter(Document.id.in_(ids)).update(
-            {Document.parent_hierarchy_node_id: desired_parent_id},
-            synchronize_session=False,
-        )
-        updated += len(ids)
-
-    if commit:
-        db_session.commit()
-    elif updated:
-        db_session.flush()
-
-    return updated
-
-
 def update_hierarchy_node_permissions(
    db_session: Session,
    raw_node_id: str,
--- a/backend/onyx/db/llm.py
+++ b/backend/onyx/db/llm.py
@@ -25,11 +25,8 @@ from onyx.server.manage.embedding.models import CloudEmbeddingProvider
 from onyx.server.manage.embedding.models import CloudEmbeddingProviderCreationRequest
 from onyx.server.manage.llm.models import LLMProviderUpsertRequest
 from onyx.server.manage.llm.models import LLMProviderView
-from onyx.utils.logger import setup_logger
 from shared_configs.enums import EmbeddingProvider

-logger = setup_logger()
-

 def update_group_llm_provider_relationships__no_commit(
    llm_provider_id: int,
@@ -112,38 +109,45 @@ def can_user_access_llm_provider(
        is_admin: If True, bypass user group restrictions but still respect persona restrictions

    Access logic:
-    - is_public controls USER access (group bypass): when True, all users can access
-      regardless of group membership. When False, user must be in a whitelisted group
-      (or be admin).
-    - Persona restrictions are ALWAYS enforced when set, regardless of is_public.
-      This allows admins to make a provider available to all users while still
-      restricting which personas (assistants) can use it.
-
-    Decision matrix:
-    1. is_public=True, no personas set → everyone has access
-    2. is_public=True, personas set → all users, but only whitelisted personas
-    3. is_public=False, groups+personas set → must satisfy BOTH (admins bypass groups)
-    4. is_public=False, only groups set → must be in group (admins bypass)
-    5. is_public=False, only personas set → must use whitelisted persona
-    6. is_public=False, neither set → admin-only (locked)
+    1. If is_public=True → everyone has access (public override)
+    2. If is_public=False:
+       - Both groups AND personas set → must satisfy BOTH (AND logic, admins bypass group check)
+       - Only groups set → must be in one of the groups (OR across groups, admins bypass)
+       - Only personas set → must use one of the personas (OR across personas, applies to admins)
+       - Neither set → NOBODY has access unless admin (locked, admin-only)
    """
-    provider_group_ids = {g.id for g in (provider.groups or [])}
-    provider_persona_ids = {p.id for p in (provider.personas or [])}
-    has_groups = bool(provider_group_ids)
-    has_personas = bool(provider_persona_ids)
-
-    # Persona restrictions are always enforced when set, regardless of is_public
-    if has_personas and not (persona and persona.id in provider_persona_ids):
-        return False
-
+    # Public override - everyone has access
    if provider.is_public:
        return True

+    # Extract IDs once to avoid multiple iterations
+    provider_group_ids = (
+        {group.id for group in provider.groups} if provider.groups else set()
+    )
+    provider_persona_ids = (
+        {p.id for p in provider.personas} if provider.personas else set()
+    )
+
+    has_groups = bool(provider_group_ids)
+    has_personas = bool(provider_persona_ids)
+
+    # Both groups AND personas set → AND logic (must satisfy both)
+    if has_groups and has_personas:
+        # Admins bypass group check but still must satisfy persona restrictions
+        user_in_group = is_admin or bool(user_group_ids & provider_group_ids)
+        persona_allowed = persona.id in provider_persona_ids if persona else False
+        return user_in_group and persona_allowed
+
+    # Only groups set → user must be in one of the groups (admins bypass)
    if has_groups:
        return is_admin or bool(user_group_ids & provider_group_ids)

-    # No groups: either persona-whitelisted (already passed) or admin-only if locked
-    return has_personas or is_admin
+    # Only personas set → persona must be in allowed list (applies to admins too)
+    if has_personas:
+        return persona.id in provider_persona_ids if persona else False
+
+    # Neither groups nor personas set, and not public → admins can access
+    return is_admin


 def validate_persona_ids_exist(
@@ -535,7 +539,6 @@ def fetch_default_model(
 ) -> ModelConfiguration | None:
    model_config = db_session.scalar(
        select(ModelConfiguration)
-        .options(selectinload(ModelConfiguration.llm_provider))
        .join(LLMModelFlow)
        .where(
            ModelConfiguration.is_visible == True,  # noqa: E712
@@ -815,43 +818,6 @@ def sync_auto_mode_models(
            changes += 1

    db_session.commit()
-
-    # Update the default if this provider currently holds the global CHAT default
-    recommended_default = llm_recommendations.get_default_model(provider.provider)
-    if recommended_default:
-        current_default_name = db_session.scalar(
-            select(ModelConfiguration.name)
-            .join(
-                LLMModelFlow,
-                LLMModelFlow.model_configuration_id == ModelConfiguration.id,
-            )
-            .where(
-                ModelConfiguration.llm_provider_id == provider.id,
-                LLMModelFlow.llm_model_flow_type == LLMModelFlowType.CHAT,
-                LLMModelFlow.is_default == True,  # noqa: E712
-            )
-        )
-
-        if (
-            current_default_name is not None
-            and current_default_name != recommended_default.name
-        ):
-            try:
-                _update_default_model(
-                    db_session=db_session,
-                    provider_id=provider.id,
-                    model=recommended_default.name,
-                    flow_type=LLMModelFlowType.CHAT,
-                )
-                changes += 1
-            except ValueError:
-                logger.warning(
-                    "Recommended default model '%s' not found "
-                    "for provider_id=%s; skipping default update.",
-                    recommended_default.name,
-                    provider.id,
-                )
-
    return changes


--- a/backend/onyx/db/models.py
+++ b/backend/onyx/db/models.py
@@ -36,11 +36,9 @@ from sqlalchemy import Text
 from sqlalchemy import text
 from sqlalchemy import UniqueConstraint
 from sqlalchemy.dialects import postgresql
-from sqlalchemy import event
 from sqlalchemy.engine.interfaces import Dialect
 from sqlalchemy.orm import DeclarativeBase
 from sqlalchemy.orm import Mapped
-from sqlalchemy.orm import Mapper
 from sqlalchemy.orm import mapped_column
 from sqlalchemy.orm import relationship
 from sqlalchemy.types import LargeBinary
@@ -105,6 +103,7 @@ from onyx.utils.encryption import encrypt_string_to_bytes
 from onyx.utils.sensitive import SensitiveValue
 from onyx.utils.headers import HeaderItemDict
 from shared_configs.enums import EmbeddingProvider
+from onyx.context.search.enums import RecencyBiasSetting

 # TODO: After anonymous user migration has been deployed, make user_id columns NOT NULL
 # and update Mapped[User | None] relationships to Mapped[User] where needed.
@@ -119,50 +118,10 @@ class Base(DeclarativeBase):
    __abstract__ = True


-class _EncryptedBase(TypeDecorator):
-    """Base for encrypted column types that wrap values in SensitiveValue."""
-
+class EncryptedString(TypeDecorator):
    impl = LargeBinary
+    # This type's behavior is fully deterministic and doesn't depend on any external factors.
    cache_ok = True
-    _is_json: bool = False
-
-    def wrap_raw(self, value: Any) -> SensitiveValue:
-        """Encrypt a raw value and wrap it in SensitiveValue.
-
-        Called by the attribute set event so the Python-side type is always
-        SensitiveValue, regardless of whether the value was loaded from the DB
-        or assigned in application code.
-        """
-        if self._is_json:
-            if not isinstance(value, dict):
-                raise TypeError(
-                    f"EncryptedJson column expected dict, got {type(value).__name__}"
-                )
-            raw_str = json.dumps(value)
-        else:
-            if not isinstance(value, str):
-                raise TypeError(
-                    f"EncryptedString column expected str, got {type(value).__name__}"
-                )
-            raw_str = value
-        return SensitiveValue(
-            encrypted_bytes=encrypt_string_to_bytes(raw_str),
-            decrypt_fn=decrypt_bytes_to_string,
-            is_json=self._is_json,
-        )
-
-    def compare_values(self, x: Any, y: Any) -> bool:
-        if x is None or y is None:
-            return x == y
-        if isinstance(x, SensitiveValue):
-            x = x.get_value(apply_mask=False)
-        if isinstance(y, SensitiveValue):
-            y = y.get_value(apply_mask=False)
-        return x == y
-
-
-class EncryptedString(_EncryptedBase):
-    _is_json: bool = False

    def process_bind_param(
        self, value: str | SensitiveValue[str] | None, dialect: Dialect  # noqa: ARG002
@@ -186,9 +145,20 @@ class EncryptedString(_EncryptedBase):
            )
        return None

+    def compare_values(self, x: Any, y: Any) -> bool:
+        if x is None or y is None:
+            return x == y
+        if isinstance(x, SensitiveValue):
+            x = x.get_value(apply_mask=False)
+        if isinstance(y, SensitiveValue):
+            y = y.get_value(apply_mask=False)
+        return x == y

-class EncryptedJson(_EncryptedBase):
-    _is_json: bool = True
+
+class EncryptedJson(TypeDecorator):
+    impl = LargeBinary
+    # This type's behavior is fully deterministic and doesn't depend on any external factors.
+    cache_ok = True

    def process_bind_param(
        self,
@@ -196,7 +166,9 @@ class EncryptedJson(_EncryptedBase):
        dialect: Dialect,  # noqa: ARG002
    ) -> bytes | None:
        if value is not None:
+            # Handle both raw dicts and SensitiveValue wrappers
            if isinstance(value, SensitiveValue):
+                # Get raw value for storage
                value = value.get_value(apply_mask=False)
            json_str = json.dumps(value)
            return encrypt_string_to_bytes(json_str)
@@ -213,40 +185,14 @@ class EncryptedJson(_EncryptedBase):
            )
        return None

-
-_REGISTERED_ATTRS: set[str] = set()
-
-
-@event.listens_for(Mapper, "mapper_configured")
-def _register_sensitive_value_set_events(
-    mapper: Mapper,
-    class_: type,
-) -> None:
-    """Auto-wrap raw values in SensitiveValue when assigned to encrypted columns."""
-    for prop in mapper.column_attrs:
-        for col in prop.columns:
-            if isinstance(col.type, _EncryptedBase):
-                col_type = col.type
-                attr = getattr(class_, prop.key)
-
-                # Guard against double-registration (e.g. if mapper is
-                # re-configured in test setups)
-                attr_key = f"{class_.__qualname__}.{prop.key}"
-                if attr_key in _REGISTERED_ATTRS:
-                    continue
-                _REGISTERED_ATTRS.add(attr_key)
-
-                @event.listens_for(attr, "set", retval=True)
-                def _wrap_value(
-                    target: Any,  # noqa: ARG001
-                    value: Any,
-                    oldvalue: Any,  # noqa: ARG001
-                    initiator: Any,  # noqa: ARG001
-                    _col_type: _EncryptedBase = col_type,
-                ) -> Any:
-                    if value is not None and not isinstance(value, SensitiveValue):
-                        return _col_type.wrap_raw(value)
-                    return value
+    def compare_values(self, x: Any, y: Any) -> bool:
+        if x is None or y is None:
+            return x == y
+        if isinstance(x, SensitiveValue):
+            x = x.get_value(apply_mask=False)
+        if isinstance(y, SensitiveValue):
+            y = y.get_value(apply_mask=False)
+        return x == y


 class NullFilteredString(TypeDecorator):
@@ -3319,6 +3265,19 @@ class Persona(Base):
    )
    name: Mapped[str] = mapped_column(String)
    description: Mapped[str] = mapped_column(String)
+    # Number of chunks to pass to the LLM for generation.
+    num_chunks: Mapped[float | None] = mapped_column(Float, nullable=True)
+    chunks_above: Mapped[int] = mapped_column(Integer)
+    chunks_below: Mapped[int] = mapped_column(Integer)
+    # Pass every chunk through LLM for evaluation, fairly expensive
+    # Can be turned off globally by admin, in which case, this setting is ignored
+    llm_relevance_filter: Mapped[bool] = mapped_column(Boolean)
+    # Enables using LLM to extract time and source type filters
+    # Can also be admin disabled globally
+    llm_filter_extraction: Mapped[bool] = mapped_column(Boolean)
+    recency_bias: Mapped[RecencyBiasSetting] = mapped_column(
+        Enum(RecencyBiasSetting, native_enum=False)
+    )

    # Allows the persona to specify a specific default LLM model
    # NOTE: only is applied on the actual response generation - is not used for things like
@@ -3345,8 +3304,11 @@ class Persona(Base):
    # Treated specially (cannot be user edited etc.)
    builtin_persona: Mapped[bool] = mapped_column(Boolean, default=False)

-    # Featured personas are highlighted in the UI
-    featured: Mapped[bool] = mapped_column(Boolean, default=False)
+    # Default personas are personas created by admins and are automatically added
+    # to all users' assistants list.
+    is_default_persona: Mapped[bool] = mapped_column(
+        Boolean, default=False, nullable=False
+    )
    # controls whether the persona is available to be selected by users
    is_visible: Mapped[bool] = mapped_column(Boolean, default=True)
    # controls the ordering of personas in the UI
@@ -4981,9 +4943,7 @@ class ScimUserMapping(Base):
    __tablename__ = "scim_user_mapping"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
-    external_id: Mapped[str | None] = mapped_column(
-        String, unique=True, index=True, nullable=True
-    )
+    external_id: Mapped[str] = mapped_column(String, unique=True, index=True)
    user_id: Mapped[UUID] = mapped_column(
        ForeignKey("user.id", ondelete="CASCADE"), unique=True, nullable=False
    )
@@ -5040,25 +5000,3 @@ class CodeInterpreterServer(Base):

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    server_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
-
-
-class CacheStore(Base):
-    """Key-value cache table used by ``PostgresCacheBackend``.
-
-    Replaces Redis for simple KV caching, locks, and list operations
-    when ``CACHE_BACKEND=postgres`` (NO_VECTOR_DB deployments).
-
-    Intentionally separate from ``KVStore``:
-    - Stores raw bytes (LargeBinary) vs JSONB, matching Redis semantics.
-    - Has ``expires_at`` for TTL; rows are periodically garbage-collected.
-    - Holds ephemeral data (tokens, stop signals, lock state) not
-      persistent application config, so cleanup can be aggressive.
-    """
-
-    __tablename__ = "cache_store"
-
-    key: Mapped[str] = mapped_column(String, primary_key=True)
-    value: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True)
-    expires_at: Mapped[datetime.datetime | None] = mapped_column(
-        DateTime(timezone=True), nullable=True
-    )
--- a/backend/onyx/db/persona.py
+++ b/backend/onyx/db/persona.py
@@ -18,8 +18,11 @@ from sqlalchemy.orm import Session
 from onyx.access.hierarchy_access import get_user_external_group_ids
 from onyx.auth.schemas import UserRole
 from onyx.configs.app_configs import CURATORS_CANNOT_VIEW_OR_EDIT_NON_OWNED_ASSISTANTS
+from onyx.configs.chat_configs import CONTEXT_CHUNKS_ABOVE
+from onyx.configs.chat_configs import CONTEXT_CHUNKS_BELOW
 from onyx.configs.constants import DEFAULT_PERSONA_ID
 from onyx.configs.constants import NotificationType
+from onyx.context.search.enums import RecencyBiasSetting
 from onyx.db.constants import SLACK_BOT_PERSONA_PREFIX
 from onyx.db.document_access import get_accessible_documents_by_ids
 from onyx.db.models import ConnectorCredentialPair
@@ -251,15 +254,13 @@ def create_update_persona(
    # Permission to actually use these is checked later

    try:
-        # Featured persona validation
-        if create_persona_request.featured:
-
-            # Curators can edit featured personas, but not make them
-            # TODO this will be reworked soon with RBAC permissions feature
+        # Default persona validation
+        if create_persona_request.is_default_persona:
+            # Curators can edit default personas, but not make them
            if user.role == UserRole.CURATOR or user.role == UserRole.GLOBAL_CURATOR:
                pass
            elif user.role != UserRole.ADMIN:
-                raise ValueError("Only admins can make a featured persona")
+                raise ValueError("Only admins can make a default persona")

        # Convert incoming string UUIDs to UUID objects for DB operations
        converted_user_file_ids = None
@@ -280,6 +281,7 @@ def create_update_persona(
            document_set_ids=create_persona_request.document_set_ids,
            tool_ids=create_persona_request.tool_ids,
            is_public=create_persona_request.is_public,
+            recency_bias=create_persona_request.recency_bias,
            llm_model_provider_override=create_persona_request.llm_model_provider_override,
            llm_model_version_override=create_persona_request.llm_model_version_override,
            starter_messages=create_persona_request.starter_messages,
@@ -293,7 +295,10 @@ def create_update_persona(
            remove_image=create_persona_request.remove_image,
            search_start_date=create_persona_request.search_start_date,
            label_ids=create_persona_request.label_ids,
-            featured=create_persona_request.featured,
+            num_chunks=create_persona_request.num_chunks,
+            llm_relevance_filter=create_persona_request.llm_relevance_filter,
+            llm_filter_extraction=create_persona_request.llm_filter_extraction,
+            is_default_persona=create_persona_request.is_default_persona,
            user_file_ids=converted_user_file_ids,
            commit=False,
            hierarchy_node_ids=create_persona_request.hierarchy_node_ids,
@@ -869,6 +874,10 @@ def upsert_persona(
    user: User | None,
    name: str,
    description: str,
+    num_chunks: float,
+    llm_relevance_filter: bool,
+    llm_filter_extraction: bool,
+    recency_bias: RecencyBiasSetting,
    llm_model_provider_override: str | None,
    llm_model_version_override: str | None,
    starter_messages: list[StarterMessage] | None,
@@ -889,11 +898,13 @@ def upsert_persona(
    remove_image: bool | None = None,
    search_start_date: datetime | None = None,
    builtin_persona: bool = False,
-    featured: bool | None = None,
+    is_default_persona: bool | None = None,
    label_ids: list[int] | None = None,
    user_file_ids: list[UUID] | None = None,
    hierarchy_node_ids: list[int] | None = None,
    document_ids: list[str] | None = None,
+    chunks_above: int = CONTEXT_CHUNKS_ABOVE,
+    chunks_below: int = CONTEXT_CHUNKS_BELOW,
    replace_base_system_prompt: bool = False,
 ) -> Persona:
    """
@@ -1004,6 +1015,12 @@ def upsert_persona(
        # `default` and `built-in` properties can only be set when creating a persona.
        existing_persona.name = name
        existing_persona.description = description
+        existing_persona.num_chunks = num_chunks
+        existing_persona.chunks_above = chunks_above
+        existing_persona.chunks_below = chunks_below
+        existing_persona.llm_relevance_filter = llm_relevance_filter
+        existing_persona.llm_filter_extraction = llm_filter_extraction
+        existing_persona.recency_bias = recency_bias
        existing_persona.llm_model_provider_override = llm_model_provider_override
        existing_persona.llm_model_version_override = llm_model_version_override
        existing_persona.starter_messages = starter_messages
@@ -1017,8 +1034,10 @@ def upsert_persona(
        if label_ids is not None:
            existing_persona.labels.clear()
            existing_persona.labels = labels or []
-        existing_persona.featured = (
-            featured if featured is not None else existing_persona.featured
+        existing_persona.is_default_persona = (
+            is_default_persona
+            if is_default_persona is not None
+            else existing_persona.is_default_persona
        )
        # Update embedded prompt fields if provided
        if system_prompt is not None:
@@ -1071,6 +1090,12 @@ def upsert_persona(
            is_public=is_public,
            name=name,
            description=description,
+            num_chunks=num_chunks,
+            chunks_above=chunks_above,
+            chunks_below=chunks_below,
+            llm_relevance_filter=llm_relevance_filter,
+            llm_filter_extraction=llm_filter_extraction,
+            recency_bias=recency_bias,
            builtin_persona=builtin_persona,
            system_prompt=system_prompt or "",
            task_prompt=task_prompt or "",
@@ -1086,7 +1111,9 @@ def upsert_persona(
            display_priority=display_priority,
            is_visible=is_visible,
            search_start_date=search_start_date,
-            featured=(featured if featured is not None else False),
+            is_default_persona=(
+                is_default_persona if is_default_persona is not None else False
+            ),
            user_files=user_files or [],
            labels=labels or [],
            hierarchy_nodes=hierarchy_nodes or [],
@@ -1131,9 +1158,9 @@ def delete_old_default_personas(
    db_session.commit()


-def update_persona_featured(
+def update_persona_is_default(
    persona_id: int,
-    featured: bool,
+    is_default: bool,
    db_session: Session,
    user: User,
 ) -> None:
@@ -1141,7 +1168,7 @@ def update_persona_featured(
        db_session=db_session, persona_id=persona_id, user=user, get_editable=True
    )

-    persona.featured = featured
+    persona.is_default_persona = is_default
    db_session.commit()


--- a/backend/onyx/db/projects.py
+++ b/backend/onyx/db/projects.py
@@ -9,9 +9,8 @@ from pydantic import BaseModel
 from pydantic import ConfigDict
 from sqlalchemy import func
 from sqlalchemy.orm import Session
-from starlette.background import BackgroundTasks

-from onyx.configs.app_configs import DISABLE_VECTOR_DB
+from onyx.background.celery.versioned_apps.client import app as client_app
 from onyx.configs.constants import FileOrigin
 from onyx.configs.constants import OnyxCeleryPriority
 from onyx.configs.constants import OnyxCeleryQueues
@@ -52,7 +51,7 @@ def create_user_files(
 ) -> CategorizedFilesResult:

    # Categorize the files
-    categorized_files = categorize_uploaded_files(files, db_session)
+    categorized_files = categorize_uploaded_files(files)
    # NOTE: At the moment, zip metadata is not used for user files.
    # Should revisit to decide whether this should be a feature.
    upload_response = upload_files(categorized_files.acceptable, FileOrigin.USER_FILE)
@@ -106,8 +105,8 @@ def upload_files_to_user_files_with_indexing(
    user: User,
    temp_id_map: dict[str, str] | None,
    db_session: Session,
-    background_tasks: BackgroundTasks | None = None,
 ) -> CategorizedFilesResult:
+    # Validate project ownership if a project_id is provided
    if project_id is not None and user is not None:
        if not check_project_ownership(project_id, user.id, db_session):
            raise HTTPException(status_code=404, detail="Project not found")
@@ -128,27 +127,16 @@ def upload_files_to_user_files_with_indexing(
        logger.warning(
            f"File {rejected_file.filename} rejected for {rejected_file.reason}"
        )
-
-    if DISABLE_VECTOR_DB and background_tasks is not None:
-        from onyx.background.task_utils import drain_processing_loop
-
-        background_tasks.add_task(drain_processing_loop, tenant_id)
-        for user_file in user_files:
-            logger.info(f"Queued in-process processing for user_file_id={user_file.id}")
-    else:
-        from onyx.background.celery.versioned_apps.client import app as client_app
-
-        for user_file in user_files:
-            task = client_app.send_task(
-                OnyxCeleryTask.PROCESS_SINGLE_USER_FILE,
-                kwargs={"user_file_id": user_file.id, "tenant_id": tenant_id},
-                queue=OnyxCeleryQueues.USER_FILE_PROCESSING,
-                priority=OnyxCeleryPriority.HIGH,
-            )
-            logger.info(
-                f"Triggered indexing for user_file_id={user_file.id} "
-                f"with task_id={task.id}"
-            )
+    for user_file in user_files:
+        task = client_app.send_task(
+            OnyxCeleryTask.PROCESS_SINGLE_USER_FILE,
+            kwargs={"user_file_id": user_file.id, "tenant_id": tenant_id},
+            queue=OnyxCeleryQueues.USER_FILE_PROCESSING,
+            priority=OnyxCeleryPriority.HIGH,
+        )
+        logger.info(
+            f"Triggered indexing for user_file_id={user_file.id} with task_id={task.id}"
+        )

    return CategorizedFilesResult(
        user_files=user_files,
--- a/backend/onyx/db/rotate_encryption_key.py
+++ b/backend/onyx/db/rotate_encryption_key.py
@@ -1,161 +0,0 @@
-"""Rotate encryption key for all encrypted columns.
-
-Dynamically discovers all columns using EncryptedString / EncryptedJson,
-decrypts each value with the old key, and re-encrypts with the current
-ENCRYPTION_KEY_SECRET.
-
-The operation is idempotent: rows already encrypted with the current key
-are skipped. Commits are made in batches so a crash mid-rotation can be
-safely resumed by re-running.
-"""
-
-import json
-from typing import Any
-
-from sqlalchemy import LargeBinary
-from sqlalchemy import select
-from sqlalchemy import update
-from sqlalchemy.orm import Session
-
-from onyx.configs.app_configs import ENCRYPTION_KEY_SECRET
-from onyx.db.models import Base
-from onyx.db.models import EncryptedJson
-from onyx.db.models import EncryptedString
-from onyx.utils.encryption import decrypt_bytes_to_string
-from onyx.utils.logger import setup_logger
-from onyx.utils.variable_functionality import global_version
-
-logger = setup_logger()
-
-_BATCH_SIZE = 500
-
-
-def _can_decrypt_with_current_key(data: bytes) -> bool:
-    """Check if data is already encrypted with the current key.
-
-    Passes the key explicitly so the fallback-to-raw-decode path in
-    _decrypt_bytes is NOT triggered — a clean success/failure signal.
-    """
-    try:
-        decrypt_bytes_to_string(data, key=ENCRYPTION_KEY_SECRET)
-        return True
-    except Exception:
-        return False
-
-
-def _discover_encrypted_columns() -> list[tuple[type, str, list[str], bool]]:
-    """Walk all ORM models and find columns using EncryptedString/EncryptedJson.
-
-    Returns list of (ModelClass, column_attr_name, [pk_attr_names], is_json).
-    """
-    results: list[tuple[type, str, list[str], bool]] = []
-
-    for mapper in Base.registry.mappers:
-        model_cls = mapper.class_
-        pk_names = [col.key for col in mapper.primary_key]
-
-        for prop in mapper.column_attrs:
-            for col in prop.columns:
-                if isinstance(col.type, EncryptedJson):
-                    results.append((model_cls, prop.key, pk_names, True))
-                elif isinstance(col.type, EncryptedString):
-                    results.append((model_cls, prop.key, pk_names, False))
-
-    return results
-
-
-def rotate_encryption_key(
-    db_session: Session,
-    old_key: str | None,
-    dry_run: bool = False,
-) -> dict[str, int]:
-    """Decrypt all encrypted columns with old_key and re-encrypt with the current key.
-
-    Args:
-        db_session: Active database session.
-        old_key: The previous encryption key. Pass None or "" if values were
-                 not previously encrypted with a key.
-        dry_run: If True, count rows that need rotation without modifying data.
-
-    Returns:
-        Dict of "table.column" -> number of rows re-encrypted (or would be).
-
-    Commits every _BATCH_SIZE rows so that locks are held briefly and progress
-    is preserved on crash. Already-rotated rows are detected and skipped,
-    making the operation safe to re-run.
-    """
-    if not global_version.is_ee_version():
-        raise RuntimeError("EE mode is not enabled — rotation requires EE encryption.")
-
-    if not ENCRYPTION_KEY_SECRET:
-        raise RuntimeError(
-            "ENCRYPTION_KEY_SECRET is not set — cannot rotate. "
-            "Set the target encryption key in the environment before running."
-        )
-
-    encrypted_columns = _discover_encrypted_columns()
-    totals: dict[str, int] = {}
-
-    for model_cls, col_name, pk_names, is_json in encrypted_columns:
-        table_name: str = model_cls.__tablename__  # type: ignore[attr-defined]
-        col_attr = getattr(model_cls, col_name)
-        pk_attrs = [getattr(model_cls, pk) for pk in pk_names]
-
-        # Read raw bytes directly, bypassing the TypeDecorator
-        raw_col = col_attr.property.columns[0]
-
-        stmt = select(*pk_attrs, raw_col.cast(LargeBinary)).where(col_attr.is_not(None))
-        rows = db_session.execute(stmt).all()
-
-        reencrypted = 0
-        batch_pending = 0
-        for row in rows:
-            raw_bytes: bytes | None = row[-1]
-            if raw_bytes is None:
-                continue
-
-            if _can_decrypt_with_current_key(raw_bytes):
-                continue
-
-            try:
-                if not old_key:
-                    decrypted_str = raw_bytes.decode("utf-8")
-                else:
-                    decrypted_str = decrypt_bytes_to_string(raw_bytes, key=old_key)
-
-                # For EncryptedJson, parse back to dict so the TypeDecorator
-                # can json.dumps() it cleanly (avoids double-encoding).
-                value: Any = json.loads(decrypted_str) if is_json else decrypted_str
-            except (ValueError, UnicodeDecodeError) as e:
-                pk_vals = [row[i] for i in range(len(pk_names))]
-                logger.warning(
-                    f"Could not decrypt/parse {table_name}.{col_name} "
-                    f"row {pk_vals} — skipping: {e}"
-                )
-                continue
-
-            if not dry_run:
-                pk_filters = [pk_attr == row[i] for i, pk_attr in enumerate(pk_attrs)]
-                update_stmt = (
-                    update(model_cls).where(*pk_filters).values({col_name: value})
-                )
-                db_session.execute(update_stmt)
-                batch_pending += 1
-
-                if batch_pending >= _BATCH_SIZE:
-                    db_session.commit()
-                    batch_pending = 0
-            reencrypted += 1
-
-        # Flush remaining rows in this column
-        if batch_pending > 0:
-            db_session.commit()
-
-        if reencrypted > 0:
-            totals[f"{table_name}.{col_name}"] = reencrypted
-            logger.info(
-                f"{'[DRY RUN] Would re-encrypt' if dry_run else 'Re-encrypted'} "
-                f"{reencrypted} value(s) in {table_name}.{col_name}"
-            )
-
-    return totals
--- a/backend/onyx/db/search_settings.py
+++ b/backend/onyx/db/search_settings.py
@@ -129,7 +129,7 @@ def get_current_search_settings(db_session: Session) -> SearchSettings:
    latest_settings = result.scalars().first()

    if not latest_settings:
-        raise RuntimeError("No search settings specified; DB is not in a valid state.")
+        raise RuntimeError("No search settings specified, DB is not in a valid state")
    return latest_settings


--- a/backend/onyx/db/slack_channel_config.py
+++ b/backend/onyx/db/slack_channel_config.py
@@ -5,6 +5,8 @@ from sqlalchemy import select
 from sqlalchemy.orm import joinedload
 from sqlalchemy.orm import Session

+from onyx.configs.chat_configs import MAX_CHUNKS_FED_TO_CHAT
+from onyx.context.search.enums import RecencyBiasSetting
 from onyx.db.constants import DEFAULT_PERSONA_SLACK_CHANNEL_NAME
 from onyx.db.constants import SLACK_BOT_PERSONA_PREFIX
 from onyx.db.models import ChannelConfig
@@ -43,6 +45,8 @@ def create_slack_channel_persona(
    channel_name: str | None,
    document_set_ids: list[int],
    existing_persona_id: int | None = None,
+    num_chunks: float = MAX_CHUNKS_FED_TO_CHAT,
+    enable_auto_filters: bool = False,
 ) -> Persona:
    """NOTE: does not commit changes"""

@@ -69,13 +73,17 @@ def create_slack_channel_persona(
        system_prompt="",
        task_prompt="",
        datetime_aware=True,
+        num_chunks=num_chunks,
+        llm_relevance_filter=True,
+        llm_filter_extraction=enable_auto_filters,
+        recency_bias=RecencyBiasSetting.AUTO,
        tool_ids=[search_tool.id],
        document_set_ids=document_set_ids,
        llm_model_provider_override=None,
        llm_model_version_override=None,
        starter_messages=None,
        is_public=True,
-        featured=False,
+        is_default_persona=False,
        db_session=db_session,
        commit=False,
    )
--- a/backend/onyx/db/tools.py
+++ b/backend/onyx/db/tools.py
@@ -13,15 +13,12 @@ from onyx.db.constants import UNSET
 from onyx.db.constants import UnsetType
 from onyx.db.enums import MCPServerStatus
 from onyx.db.models import MCPServer
-from onyx.db.models import OAuthConfig
 from onyx.db.models import Tool
 from onyx.db.models import ToolCall
 from onyx.server.features.tool.models import Header
 from onyx.tools.built_in_tools import BUILT_IN_TOOL_TYPES
 from onyx.utils.headers import HeaderItemDict
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_json_like
-from onyx.utils.postgres_sanitization import sanitize_string

 if TYPE_CHECKING:
    pass
@@ -162,26 +159,10 @@ def update_tool(
        ]
    if passthrough_auth is not None:
        tool.passthrough_auth = passthrough_auth
-    old_oauth_config_id = tool.oauth_config_id
    if not isinstance(oauth_config_id, UnsetType):
        tool.oauth_config_id = oauth_config_id
-        db_session.flush()
-
-    # Clean up orphaned OAuthConfig if the oauth_config_id was changed
-    if (
-        old_oauth_config_id is not None
-        and not isinstance(oauth_config_id, UnsetType)
-        and old_oauth_config_id != oauth_config_id
-    ):
-        other_tools = db_session.scalars(
-            select(Tool).where(Tool.oauth_config_id == old_oauth_config_id)
-        ).all()
-        if not other_tools:
-            oauth_config = db_session.get(OAuthConfig, old_oauth_config_id)
-            if oauth_config:
-                db_session.delete(oauth_config)
-
    db_session.commit()
+
    return tool


@@ -190,21 +171,8 @@ def delete_tool__no_commit(tool_id: int, db_session: Session) -> None:
    if tool is None:
        raise ValueError(f"Tool with ID {tool_id} does not exist")

-    oauth_config_id = tool.oauth_config_id
-
    db_session.delete(tool)
-    db_session.flush()
-
-    # Clean up orphaned OAuthConfig if no other tools reference it
-    if oauth_config_id is not None:
-        other_tools = db_session.scalars(
-            select(Tool).where(Tool.oauth_config_id == oauth_config_id)
-        ).all()
-        if not other_tools:
-            oauth_config = db_session.get(OAuthConfig, oauth_config_id)
-            if oauth_config:
-                db_session.delete(oauth_config)
-                db_session.flush()
+    db_session.flush()  # Don't commit yet, let caller decide when to commit


 def get_builtin_tool(
@@ -288,13 +256,11 @@ def create_tool_call_no_commit(
        tab_index=tab_index,
        tool_id=tool_id,
        tool_call_id=tool_call_id,
-        reasoning_tokens=(
-            sanitize_string(reasoning_tokens) if reasoning_tokens else reasoning_tokens
-        ),
-        tool_call_arguments=sanitize_json_like(tool_call_arguments),
-        tool_call_response=sanitize_json_like(tool_call_response),
+        reasoning_tokens=reasoning_tokens,
+        tool_call_arguments=tool_call_arguments,
+        tool_call_response=tool_call_response,
        tool_call_tokens=tool_call_tokens,
-        generated_images=sanitize_json_like(generated_images),
+        generated_images=generated_images,
    )

    db_session.add(tool_call)
--- a/backend/onyx/document_index/FILTER_SEMANTICS.md
+++ b/backend/onyx/document_index/FILTER_SEMANTICS.md
@@ -1,103 +0,0 @@
-# Vector DB Filter Semantics
-
-How `IndexFilters` fields combine into the final query filter. Applies to both Vespa and OpenSearch.
-
-## Filter categories
-
-| Category | Fields | Join logic |
-|---|---|---|
-| **Visibility** | `hidden` | Always applied (unless `include_hidden`) |
-| **Tenant** | `tenant_id` | AND (multi-tenant only) |
-| **ACL** | `access_control_list` | OR within, AND with rest |
-| **Narrowing** | `source_type`, `tags`, `time_cutoff` | Each OR within, AND with rest |
-| **Knowledge scope** | `document_set`, `user_file_ids`, `attached_document_ids`, `hierarchy_node_ids` | OR within group, AND with rest |
-| **Additive scope** | `project_id`, `persona_id` | OR'd into knowledge scope **only when** a knowledge scope filter already exists |
-
-## How filters combine
-
-All categories are AND'd together. Within the knowledge scope category, individual filters are OR'd.
-
-```
-NOT hidden
-AND tenant = T                          -- if multi-tenant
-AND (acl contains A1 OR acl contains A2)
-AND (source_type = S1 OR ...)           -- if set
-AND (tag = T1 OR ...)                   -- if set
-AND <knowledge scope>                   -- see below
-AND time >= cutoff                      -- if set
-```
-
-## Knowledge scope rules
-
-The knowledge scope filter controls **what knowledge an assistant can access**.
-
-### No explicit knowledge attached
-
-When `document_set`, `user_file_ids`, `attached_document_ids`, and `hierarchy_node_ids` are all empty/None:
-
- **No knowledge scope filter is applied.** The assistant can see everything (subject to ACL).
- `project_id` and `persona_id` are ignored — they never restrict on their own.
-
-### One explicit knowledge type
-
-```
-- Only document sets
-AND (document_sets contains "Engineering" OR document_sets contains "Legal")
-
-- Only user files
-AND (document_id = "uuid-1" OR document_id = "uuid-2")
-```
-
-### Multiple explicit knowledge types (OR'd)
-
-```
-- Document sets + user files
-AND (
-    document_sets contains "Engineering"
-    OR document_id = "uuid-1"
-)
-```
-
-### Explicit knowledge + overflowing user files
-
-When an explicit knowledge restriction is in effect **and** `project_id` or `persona_id` is set (user files overflowed the LLM context window), the additive scopes widen the filter:
-
-```
-- Document sets + persona user files overflowed
-AND (
-    document_sets contains "Engineering"
-    OR personas contains 42
-)
-
-- User files + project files overflowed
-AND (
-    document_id = "uuid-1"
-    OR user_project contains 7
-)
-```
-
-### Only project_id or persona_id (no explicit knowledge)
-
-No knowledge scope filter. The assistant searches everything.
-
-```
-- Just ACL, no restriction
-NOT hidden
-AND (acl contains ...)
-```
-
-## Field reference
-
-| Filter field | Vespa field | Vespa type | Purpose |
-|---|---|---|---|
-| `document_set` | `document_sets` | `weightedset<string>` | Connector doc sets attached to assistant |
-| `user_file_ids` | `document_id` | `string` | User files uploaded to assistant |
-| `attached_document_ids` | `document_id` | `string` | Documents explicitly attached (OpenSearch only) |
-| `hierarchy_node_ids` | `ancestor_hierarchy_node_ids` | `array<int>` | Folder/space nodes (OpenSearch only) |
-| `project_id` | `user_project` | `array<int>` | Project tag for overflowing user files |
-| `persona_id` | `personas` | `array<int>` | Persona tag for overflowing user files |
-| `access_control_list` | `access_control_list` | `weightedset<string>` | ACL entries for the requesting user |
-| `source_type` | `source_type` | `string` | Connector source type (e.g. `web`, `jira`) |
-| `tags` | `metadata_list` | `array<string>` | Document metadata tags |
-| `time_cutoff` | `doc_updated_at` | `long` | Minimum document update timestamp |
-| `tenant_id` | `tenant_id` | `string` | Tenant isolation (multi-tenant) |
--- a/backend/onyx/document_index/document_index_utils.py
+++ b/backend/onyx/document_index/document_index_utils.py
@@ -32,6 +32,9 @@ def get_multipass_config(search_settings: SearchSettings) -> MultipassConfig:
    Determines whether to enable multipass and large chunks by examining
    the current search settings and the embedder configuration.
    """
+    if not search_settings:
+        return MultipassConfig(multipass_indexing=False, enable_large_chunks=False)
+
    multipass = should_use_multipass(search_settings)
    enable_large_chunks = SearchSettings.can_use_large_chunks(
        multipass, search_settings.model_name, search_settings.provider_type
--- a/backend/onyx/document_index/factory.py
+++ b/backend/onyx/document_index/factory.py
@@ -26,10 +26,11 @@ def get_default_document_index(
    To be used for retrieval only. Indexing should be done through both indices
    until Vespa is deprecated.

+    Pre-existing docstring for this function, although secondary indices are not
+    currently supported:
    Primary index is the index that is used for querying/updating etc. Secondary
    index is for when both the currently used index and the upcoming index both
-    need to be updated. Updates are applied to both indices.
-    WARNING: In that case, get_all_document_indices should be used.
+    need to be updated, updates are applied to both indices.
    """
    if DISABLE_VECTOR_DB:
        return DisabledDocumentIndex(
@@ -50,26 +51,11 @@ def get_default_document_index(
    opensearch_retrieval_enabled = get_opensearch_retrieval_state(db_session)
    if opensearch_retrieval_enabled:
        indexing_setting = IndexingSetting.from_db_model(search_settings)
-        secondary_indexing_setting = (
-            IndexingSetting.from_db_model(secondary_search_settings)
-            if secondary_search_settings
-            else None
-        )
        return OpenSearchOldDocumentIndex(
            index_name=search_settings.index_name,
            embedding_dim=indexing_setting.final_embedding_dim,
            embedding_precision=indexing_setting.embedding_precision,
            secondary_index_name=secondary_index_name,
-            secondary_embedding_dim=(
-                secondary_indexing_setting.final_embedding_dim
-                if secondary_indexing_setting
-                else None
-            ),
-            secondary_embedding_precision=(
-                secondary_indexing_setting.embedding_precision
-                if secondary_indexing_setting
-                else None
-            ),
            large_chunks_enabled=search_settings.large_chunks_enabled,
            secondary_large_chunks_enabled=secondary_large_chunks_enabled,
            multitenant=MULTI_TENANT,
@@ -100,7 +86,8 @@ def get_all_document_indices(
    Used for indexing only. Until Vespa is deprecated we will index into both
    document indices. Retrieval is done through only one index however.

-    Large chunks are not currently supported so we hardcode appropriate values.
+    Large chunks and secondary indices are not currently supported so we
+    hardcode appropriate values.

    NOTE: Make sure the Vespa index object is returned first. In the rare event
    that there is some conflict between indexing and the migration task, it is
@@ -136,36 +123,13 @@ def get_all_document_indices(
    opensearch_document_index: OpenSearchOldDocumentIndex | None = None
    if ENABLE_OPENSEARCH_INDEXING_FOR_ONYX:
        indexing_setting = IndexingSetting.from_db_model(search_settings)
-        secondary_indexing_setting = (
-            IndexingSetting.from_db_model(secondary_search_settings)
-            if secondary_search_settings
-            else None
-        )
        opensearch_document_index = OpenSearchOldDocumentIndex(
            index_name=search_settings.index_name,
            embedding_dim=indexing_setting.final_embedding_dim,
            embedding_precision=indexing_setting.embedding_precision,
-            secondary_index_name=(
-                secondary_search_settings.index_name
-                if secondary_search_settings
-                else None
-            ),
-            secondary_embedding_dim=(
-                secondary_indexing_setting.final_embedding_dim
-                if secondary_indexing_setting
-                else None
-            ),
-            secondary_embedding_precision=(
-                secondary_indexing_setting.embedding_precision
-                if secondary_indexing_setting
-                else None
-            ),
-            large_chunks_enabled=search_settings.large_chunks_enabled,
-            secondary_large_chunks_enabled=(
-                secondary_search_settings.large_chunks_enabled
-                if secondary_search_settings
-                else None
-            ),
+            secondary_index_name=None,
+            large_chunks_enabled=False,
+            secondary_large_chunks_enabled=None,
            multitenant=MULTI_TENANT,
            httpx_client=httpx_client,
        )
--- a/backend/onyx/document_index/opensearch/client.py
+++ b/backend/onyx/document_index/opensearch/client.py
@@ -61,25 +61,6 @@ class SearchHit(BaseModel, Generic[SchemaDocumentModel]):
    explanation: dict[str, Any] | None = None


-class IndexInfo(BaseModel):
-    """
-    Represents information about an OpenSearch index.
-    """
-
-    model_config = {"frozen": True}
-
-    name: str
-    health: str
-    status: str
-    num_primary_shards: str
-    num_replica_shards: str
-    docs_count: str
-    docs_deleted: str
-    created_at: str
-    total_size: str
-    primary_shards_size: str
-
-
 def get_new_body_without_vectors(body: dict[str, Any]) -> dict[str, Any]:
    """Recursively replaces vectors in the body with their length.

@@ -178,8 +159,8 @@ class OpenSearchClient(AbstractContextManager):
        Raises:
            Exception: There was an error creating the search pipeline.
        """
-        response = self._client.search_pipeline.put(id=pipeline_id, body=pipeline_body)
-        if not response.get("acknowledged", False):
+        result = self._client.search_pipeline.put(id=pipeline_id, body=pipeline_body)
+        if not result.get("acknowledged", False):
            raise RuntimeError(f"Failed to create search pipeline {pipeline_id}.")

    @log_function_time(print_only=True, debug_only=True, include_args=True)
@@ -192,8 +173,8 @@ class OpenSearchClient(AbstractContextManager):
        Raises:
            Exception: There was an error deleting the search pipeline.
        """
-        response = self._client.search_pipeline.delete(id=pipeline_id)
-        if not response.get("acknowledged", False):
+        result = self._client.search_pipeline.delete(id=pipeline_id)
+        if not result.get("acknowledged", False):
            raise RuntimeError(f"Failed to delete search pipeline {pipeline_id}.")

    @log_function_time(print_only=True, debug_only=True, include_args=True)
@@ -217,34 +198,6 @@ class OpenSearchClient(AbstractContextManager):
            logger.error(f"Failed to put cluster settings: {response}.")
            return False

-    @log_function_time(print_only=True, debug_only=True)
-    def list_indices_with_info(self) -> list[IndexInfo]:
-        """
-        Lists the indices in the OpenSearch cluster with information about each
-        index.
-
-        Returns:
-            A list of IndexInfo objects for each index.
-        """
-        response = self._client.cat.indices(format="json")
-        indices: list[IndexInfo] = []
-        for raw_index_info in response:
-            indices.append(
-                IndexInfo(
-                    name=raw_index_info.get("index", ""),
-                    health=raw_index_info.get("health", ""),
-                    status=raw_index_info.get("status", ""),
-                    num_primary_shards=raw_index_info.get("pri", ""),
-                    num_replica_shards=raw_index_info.get("rep", ""),
-                    docs_count=raw_index_info.get("docs.count", ""),
-                    docs_deleted=raw_index_info.get("docs.deleted", ""),
-                    created_at=raw_index_info.get("creation.date.string", ""),
-                    total_size=raw_index_info.get("store.size", ""),
-                    primary_shards_size=raw_index_info.get("pri.store.size", ""),
-                )
-            )
-        return indices
-
    @log_function_time(print_only=True, debug_only=True)
    def ping(self) -> bool:
        """Pings the OpenSearch cluster.
--- a/backend/onyx/document_index/opensearch/opensearch_document_index.py
+++ b/backend/onyx/document_index/opensearch/opensearch_document_index.py
@@ -6,6 +6,7 @@ import httpx
 from opensearchpy import NotFoundError

 from onyx.access.models import DocumentAccess
+from onyx.configs.app_configs import USING_AWS_MANAGED_OPENSEARCH
 from onyx.configs.app_configs import VERIFY_CREATE_OPENSEARCH_INDEX_ON_INIT_MT
 from onyx.configs.chat_configs import NUM_RETURNED_HITS
 from onyx.configs.chat_configs import TITLE_CONTENT_RATIO
@@ -271,9 +272,6 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):
        embedding_dim: int,
        embedding_precision: EmbeddingPrecision,
        secondary_index_name: str | None,
-        secondary_embedding_dim: int | None,
-        secondary_embedding_precision: EmbeddingPrecision | None,
-        # NOTE: We do not support large chunks right now.
        large_chunks_enabled: bool,  # noqa: ARG002
        secondary_large_chunks_enabled: bool | None,  # noqa: ARG002
        multitenant: bool = False,
@@ -289,25 +287,12 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):
                f"Expected {MULTI_TENANT}, got {multitenant}."
            )
        tenant_id = get_current_tenant_id()
-        tenant_state = TenantState(tenant_id=tenant_id, multitenant=multitenant)
        self._real_index = OpenSearchDocumentIndex(
-            tenant_state=tenant_state,
+            tenant_state=TenantState(tenant_id=tenant_id, multitenant=multitenant),
            index_name=index_name,
            embedding_dim=embedding_dim,
            embedding_precision=embedding_precision,
        )
-        self._secondary_real_index: OpenSearchDocumentIndex | None = None
-        if self.secondary_index_name:
-            if secondary_embedding_dim is None or secondary_embedding_precision is None:
-                raise ValueError(
-                    "Bug: Secondary index embedding dimension and precision are not set."
-                )
-            self._secondary_real_index = OpenSearchDocumentIndex(
-                tenant_state=tenant_state,
-                index_name=self.secondary_index_name,
-                embedding_dim=secondary_embedding_dim,
-                embedding_precision=secondary_embedding_precision,
-            )

    @staticmethod
    def register_multitenant_indices(
@@ -323,38 +308,19 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):
        self,
        primary_embedding_dim: int,
        primary_embedding_precision: EmbeddingPrecision,
-        secondary_index_embedding_dim: int | None,
-        secondary_index_embedding_precision: EmbeddingPrecision | None,
+        secondary_index_embedding_dim: int | None,  # noqa: ARG002
+        secondary_index_embedding_precision: EmbeddingPrecision | None,  # noqa: ARG002
    ) -> None:
-        self._real_index.verify_and_create_index_if_necessary(
+        # Only handle primary index for now, ignore secondary.
+        return self._real_index.verify_and_create_index_if_necessary(
            primary_embedding_dim, primary_embedding_precision
        )
-        if self.secondary_index_name:
-            if (
-                secondary_index_embedding_dim is None
-                or secondary_index_embedding_precision is None
-            ):
-                raise ValueError(
-                    "Bug: Secondary index embedding dimension and precision are not set."
-                )
-            assert (
-                self._secondary_real_index is not None
-            ), "Bug: Secondary index is not initialized."
-            self._secondary_real_index.verify_and_create_index_if_necessary(
-                secondary_index_embedding_dim, secondary_index_embedding_precision
-            )

    def index(
        self,
        chunks: list[DocMetadataAwareIndexChunk],
        index_batch_params: IndexBatchParams,
    ) -> set[OldDocumentInsertionRecord]:
-        """
-        NOTE: Do NOT consider the secondary index here. A separate indexing
-        pipeline will be responsible for indexing to the secondary index. This
-        design is not ideal and we should reconsider this when revamping index
-        swapping.
-        """
        # Convert IndexBatchParams to IndexingMetadata.
        chunk_counts: dict[str, IndexingMetadata.ChunkCounts] = {}
        for doc_id in index_batch_params.doc_id_to_new_chunk_cnt:
@@ -386,20 +352,7 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):
        tenant_id: str,  # noqa: ARG002
        chunk_count: int | None,
    ) -> int:
-        """
-        NOTE: Remember to handle the secondary index here. There is no separate
-        pipeline for deleting chunks in the secondary index. This design is not
-        ideal and we should reconsider this when revamping index swapping.
-        """
-        total_chunks_deleted = self._real_index.delete(doc_id, chunk_count)
-        if self.secondary_index_name:
-            assert (
-                self._secondary_real_index is not None
-            ), "Bug: Secondary index is not initialized."
-            total_chunks_deleted += self._secondary_real_index.delete(
-                doc_id, chunk_count
-            )
-        return total_chunks_deleted
+        return self._real_index.delete(doc_id, chunk_count)

    def update_single(
        self,
@@ -410,11 +363,6 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):
        fields: VespaDocumentFields | None,
        user_fields: VespaDocumentUserFields | None,
    ) -> None:
-        """
-        NOTE: Remember to handle the secondary index here. There is no separate
-        pipeline for updating chunks in the secondary index. This design is not
-        ideal and we should reconsider this when revamping index swapping.
-        """
        if fields is None and user_fields is None:
            logger.warning(
                f"Tried to update document {doc_id} with no updated fields or user fields."
@@ -445,11 +393,6 @@ class OpenSearchOldDocumentIndex(OldDocumentIndex):

        try:
            self._real_index.update([update_request])
-            if self.secondary_index_name:
-                assert (
-                    self._secondary_real_index is not None
-                ), "Bug: Secondary index is not initialized."
-                self._secondary_real_index.update([update_request])
        except NotFoundError:
            logger.exception(
                f"Tried to update document {doc_id} but at least one of its chunks was not found in OpenSearch. "
@@ -620,7 +563,12 @@ class OpenSearchDocumentIndex(DocumentIndex):
        )

        if not self._client.index_exists():
-            index_settings = DocumentSchema.get_index_settings_based_on_environment()
+            if USING_AWS_MANAGED_OPENSEARCH:
+                index_settings = (
+                    DocumentSchema.get_index_settings_for_aws_managed_opensearch()
+                )
+            else:
+                index_settings = DocumentSchema.get_index_settings()
            self._client.create_index(
                mappings=expected_mappings,
                settings=index_settings,
@@ -739,8 +687,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
            The number of chunks successfully deleted.
        """
        logger.debug(
-            f"[OpenSearchDocumentIndex] Deleting document {document_id} from index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Deleting document {document_id} from index {self._index_name}."
        )
        query_body = DocumentQuery.delete_from_document_id_query(
            document_id=document_id,
@@ -776,8 +723,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
                specified documents.
        """
        logger.debug(
-            f"[OpenSearchDocumentIndex] Updating {len(update_requests)} chunks for index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Updating {len(update_requests)} chunks for index {self._index_name}."
        )
        for update_request in update_requests:
            properties_to_update: dict[str, Any] = dict()
@@ -833,11 +779,9 @@ class OpenSearchDocumentIndex(DocumentIndex):
                    # here.
                    # TODO(andrei): Fix the aforementioned race condition.
                    raise ChunkCountNotFoundError(
-                        f"Tried to update document {doc_id} but its chunk count is not known. "
-                        "Older versions of the application used to permit this but is not a "
-                        "supported state for a document when using OpenSearch. The document was "
-                        "likely just added to the indexing pipeline and the chunk count will be "
-                        "updated shortly."
+                        f"Tried to update document {doc_id} but its chunk count is not known. Older versions of the "
+                        "application used to permit this but is not a supported state for a document when using OpenSearch. "
+                        "The document was likely just added to the indexing pipeline and the chunk count will be updated shortly."
                    )
                if doc_chunk_count == 0:
                    raise ValueError(
@@ -869,8 +813,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
        chunk IDs vs querying for matching document chunks.
        """
        logger.debug(
-            f"[OpenSearchDocumentIndex] Retrieving {len(chunk_requests)} chunks for index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Retrieving {len(chunk_requests)} chunks for index {self._index_name}."
        )
        results: list[InferenceChunk] = []
        for chunk_request in chunk_requests:
@@ -917,8 +860,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
        num_to_retrieve: int,
    ) -> list[InferenceChunk]:
        logger.debug(
-            f"[OpenSearchDocumentIndex] Hybrid retrieving {num_to_retrieve} chunks for index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Hybrid retrieving {num_to_retrieve} chunks for index {self._index_name}."
        )
        # TODO(andrei): This could be better, the caller should just make this
        # decision when passing in the query param. See the above comment in the
@@ -938,10 +880,8 @@ class OpenSearchDocumentIndex(DocumentIndex):
            index_filters=filters,
            include_hidden=False,
        )
-        # NOTE: Using z-score normalization here because it's better for hybrid
-        # search from a theoretical standpoint. Empirically on a small dataset
-        # of up to 10K docs, it's not very different. Likely more impactful at
-        # scale.
+        # NOTE: Using z-score normalization here because it's better for hybrid search from a theoretical standpoint.
+        # Empirically on a small dataset of up to 10K docs, it's not very different. Likely more impactful at scale.
        # https://opensearch.org/blog/introducing-the-z-score-normalization-technique-for-hybrid-search/
        search_hits: list[SearchHit[DocumentChunk]] = self._client.search(
            body=query_body,
@@ -968,8 +908,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
        dirty: bool | None = None,  # noqa: ARG002
    ) -> list[InferenceChunk]:
        logger.debug(
-            f"[OpenSearchDocumentIndex] Randomly retrieving {num_to_retrieve} chunks for index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Randomly retrieving {num_to_retrieve} chunks for index {self._index_name}."
        )
        query_body = DocumentQuery.get_random_search_query(
            tenant_state=self._tenant_state,
@@ -999,8 +938,7 @@ class OpenSearchDocumentIndex(DocumentIndex):
        complete.
        """
        logger.debug(
-            f"[OpenSearchDocumentIndex] Indexing {len(chunks)} raw chunks for index "
-            f"{self._index_name}."
+            f"[OpenSearchDocumentIndex] Indexing {len(chunks)} raw chunks for index {self._index_name}."
        )
        # Do not raise if the document already exists, just update. This is
        # because the document may already have been indexed during the
--- a/backend/onyx/document_index/opensearch/schema.py
+++ b/backend/onyx/document_index/opensearch/schema.py
@@ -12,7 +12,6 @@ from pydantic import model_validator
 from pydantic import SerializerFunctionWrapHandler

 from onyx.configs.app_configs import OPENSEARCH_TEXT_ANALYZER
-from onyx.configs.app_configs import USING_AWS_MANAGED_OPENSEARCH
 from onyx.document_index.interfaces_new import TenantState
 from onyx.document_index.opensearch.constants import DEFAULT_MAX_CHUNK_SIZE
 from onyx.document_index.opensearch.constants import EF_CONSTRUCTION
@@ -243,8 +242,7 @@ class DocumentChunk(BaseModel):
            return value
        if not isinstance(value, int):
            raise ValueError(
-                f"Bug: Expected an int for the last_updated property from OpenSearch, got "
-                f"{type(value)} instead."
+                f"Bug: Expected an int for the last_updated property from OpenSearch, got {type(value)} instead."
            )
        return datetime.fromtimestamp(value, tz=timezone.utc)

@@ -285,22 +283,19 @@ class DocumentChunk(BaseModel):
        elif isinstance(value, TenantState):
            if MULTI_TENANT != value.multitenant:
                raise ValueError(
-                    f"Bug: An existing TenantState object was supplied to the DocumentChunk model "
-                    f"but its multi-tenant mode ({value.multitenant}) does not match the program's "
-                    "current global tenancy state."
+                    f"Bug: An existing TenantState object was supplied to the DocumentChunk model but its multi-tenant mode "
+                    f"({value.multitenant}) does not match the program's current global tenancy state."
                )
            return value
        elif not isinstance(value, str):
            raise ValueError(
-                f"Bug: Expected a str for the tenant_id property from OpenSearch, got "
-                f"{type(value)} instead."
+                f"Bug: Expected a str for the tenant_id property from OpenSearch, got {type(value)} instead."
            )
        else:
            if not MULTI_TENANT:
                raise ValueError(
-                    "Bug: Got a non-null str for the tenant_id property from OpenSearch but "
-                    "multi-tenant mode is not enabled. This is unexpected because in single-tenant "
-                    "mode we don't expect to see a tenant_id."
+                    "Bug: Got a non-null str for the tenant_id property from OpenSearch but multi-tenant mode is not enabled. "
+                    "This is unexpected because in single-tenant mode we don't expect to see a tenant_id."
                )
            return TenantState(tenant_id=value, multitenant=MULTI_TENANT)

@@ -356,10 +351,8 @@ class DocumentSchema:
            "properties": {
                TITLE_FIELD_NAME: {
                    "type": "text",
-                    # Language analyzer (e.g. english) stems at index and search
-                    # time for variant matching. Configure via
-                    # OPENSEARCH_TEXT_ANALYZER. Existing indices need reindexing
-                    # after a change.
+                    # Language analyzer (e.g. english) stems at index and search time for variant matching.
+                    # Configure via OPENSEARCH_TEXT_ANALYZER. Existing indices need reindexing after a change.
                    "analyzer": OPENSEARCH_TEXT_ANALYZER,
                    "fields": {
                        # Subfield accessed as title.keyword. Not indexed for
@@ -532,7 +525,7 @@ class DocumentSchema:
        }

    @staticmethod
-    def get_index_settings_for_aws_managed_opensearch_st_dev() -> dict[str, Any]:
+    def get_index_settings_for_aws_managed_opensearch() -> dict[str, Any]:
        """
        Settings for AWS-managed OpenSearch.

@@ -553,41 +546,3 @@ class DocumentSchema:
                "knn.algo_param.ef_search": EF_SEARCH,
            }
        }
-
-    @staticmethod
-    def get_index_settings_for_aws_managed_opensearch_mt_cloud() -> dict[str, Any]:
-        """
-        Settings for AWS-managed OpenSearch in multi-tenant cloud.
-
-        324 shards very roughly targets a storage load of ~30Gb per shard, which
-        according to AWS OpenSearch documentation is within a good target range.
-
-        As documented above we need 2 replicas for a total of 3 copies of the
-        data because the cluster is configured with 3-AZ awareness.
-        """
-        return {
-            "index": {
-                "number_of_shards": 324,
-                "number_of_replicas": 2,
-                # Required for vector search.
-                "knn": True,
-                "knn.algo_param.ef_search": EF_SEARCH,
-            }
-        }
-
-    @staticmethod
-    def get_index_settings_based_on_environment() -> dict[str, Any]:
-        """
-        Returns the index settings based on the environment.
-        """
-        if USING_AWS_MANAGED_OPENSEARCH:
-            if MULTI_TENANT:
-                return (
-                    DocumentSchema.get_index_settings_for_aws_managed_opensearch_mt_cloud()
-                )
-            else:
-                return (
-                    DocumentSchema.get_index_settings_for_aws_managed_opensearch_st_dev()
-                )
-        else:
-            return DocumentSchema.get_index_settings()
--- a/backend/onyx/document_index/opensearch/search.py
+++ b/backend/onyx/document_index/opensearch/search.py
@@ -698,6 +698,41 @@ class DocumentQuery:
            """
            return {"terms": {ANCESTOR_HIERARCHY_NODE_IDS_FIELD_NAME: node_ids}}

+        def _get_assistant_knowledge_filter(
+            attached_doc_ids: list[str] | None,
+            node_ids: list[int] | None,
+            file_ids: list[UUID] | None,
+            document_sets: list[str] | None,
+        ) -> dict[str, Any]:
+            """Combined filter for assistant knowledge.
+
+            When an assistant has attached knowledge, search should be scoped to:
+            - Documents explicitly attached (by document ID), OR
+            - Documents under attached hierarchy nodes (by ancestor node IDs), OR
+            - User-uploaded files attached to the assistant, OR
+            - Documents in the assistant's document sets (if any)
+            """
+            knowledge_filter: dict[str, Any] = {
+                "bool": {"should": [], "minimum_should_match": 1}
+            }
+            if attached_doc_ids:
+                knowledge_filter["bool"]["should"].append(
+                    _get_attached_document_id_filter(attached_doc_ids)
+                )
+            if node_ids:
+                knowledge_filter["bool"]["should"].append(
+                    _get_hierarchy_node_filter(node_ids)
+                )
+            if file_ids:
+                knowledge_filter["bool"]["should"].append(
+                    _get_user_file_id_filter(file_ids)
+                )
+            if document_sets:
+                knowledge_filter["bool"]["should"].append(
+                    _get_document_set_filter(document_sets)
+                )
+            return knowledge_filter
+
        filter_clauses: list[dict[str, Any]] = []

        if not include_hidden:
@@ -723,53 +758,41 @@ class DocumentQuery:
            # document's metadata list.
            filter_clauses.append(_get_tag_filter(tags))

-        # Knowledge scope: explicit knowledge attachments restrict what
-        # an assistant can see.  When none are set the assistant
-        # searches everything.
-        #
-        # project_id / persona_id are additive: they make overflowing
-        # user files findable but must NOT trigger the restriction on
-        # their own (an agent with no explicit knowledge should search
-        # everything).
-        has_knowledge_scope = (
+        # Check if this is an assistant knowledge search (has any assistant-scoped knowledge)
+        has_assistant_knowledge = (
            attached_document_ids
            or hierarchy_node_ids
            or user_file_ids
            or document_sets
        )

-        if has_knowledge_scope:
-            knowledge_filter: dict[str, Any] = {
-                "bool": {"should": [], "minimum_should_match": 1}
-            }
-            if attached_document_ids:
-                knowledge_filter["bool"]["should"].append(
-                    _get_attached_document_id_filter(attached_document_ids)
+        if has_assistant_knowledge:
+            # If assistant has attached knowledge, scope search to that knowledge.
+            # Document sets are included in the OR filter so directly attached
+            # docs are always findable even if not in the document sets.
+            filter_clauses.append(
+                _get_assistant_knowledge_filter(
+                    attached_document_ids,
+                    hierarchy_node_ids,
+                    user_file_ids,
+                    document_sets,
                )
-            if hierarchy_node_ids:
-                knowledge_filter["bool"]["should"].append(
-                    _get_hierarchy_node_filter(hierarchy_node_ids)
-                )
-            if user_file_ids:
-                knowledge_filter["bool"]["should"].append(
-                    _get_user_file_id_filter(user_file_ids)
-                )
-            if document_sets:
-                knowledge_filter["bool"]["should"].append(
-                    _get_document_set_filter(document_sets)
-                )
-            # Additive: widen scope to also cover overflowing user
-            # files, but only when an explicit restriction is already
-            # in effect.
-            if project_id is not None:
-                knowledge_filter["bool"]["should"].append(
-                    _get_user_project_filter(project_id)
-                )
-            if persona_id is not None:
-                knowledge_filter["bool"]["should"].append(
-                    _get_persona_filter(persona_id)
-                )
-            filter_clauses.append(knowledge_filter)
+            )
+        elif user_file_ids:
+            # Fallback for non-assistant user file searches (e.g., project searches)
+            # If at least one user file ID is provided, the caller will only
+            # retrieve documents where the document ID is in this input list of
+            # file IDs.
+            filter_clauses.append(_get_user_file_id_filter(user_file_ids))
+
+        if project_id is not None:
+            # If a project ID is provided, the caller will only retrieve
+            # documents where the project ID provided here is present in the
+            # document's user projects list.
+            filter_clauses.append(_get_user_project_filter(project_id))
+
+        if persona_id is not None:
+            filter_clauses.append(_get_persona_filter(persona_id))

        if time_cutoff is not None:
            # If a time cutoff is provided, the caller will only retrieve
--- a/Show More
+++ b/Show More