Hide deprecated columns

.cursor/skills/onyx-cli/SKILL.md

@@ -1,186 +0,0 @@
----
-name: onyx-cli
-description: Query the Onyx knowledge base using the onyx-cli command. Use when the user wants to search company documents, ask questions about internal knowledge, query connected data sources, or look up information stored in Onyx.
----
-
-# Onyx CLI — Agent Tool
-
-Onyx is an enterprise search and Gen-AI platform that connects to company documents, apps, and people. The `onyx-cli` CLI provides non-interactive commands to query the Onyx knowledge base and list available agents.
-
-## Prerequisites
-
-### 1. Check if installed
-
-```bash
-which onyx-cli
-```
-
-### 2. Install (if needed)
-
-**Primary — pip:**
-
-```bash
-pip install onyx-cli
-```
-
-**From source (Go):**
-
-```bash
-cd cli && go build -o onyx-cli . && sudo mv onyx-cli /usr/local/bin/
-```
-
-### 3. Check if configured
-
-```bash
-onyx-cli validate-config
-```
-
-This checks the config file exists, API key is present, and tests the server connection via `/api/me`. Exit code 0 on success, non-zero with a descriptive error on failure.
-
-If unconfigured, you have two options:
-
-**Option A — Interactive setup (requires user input):**
-
-```bash
-onyx-cli configure
-```
-
-This prompts for the Onyx server URL and API key, tests the connection, and saves config.
-
-**Option B — Environment variables (non-interactive, preferred for agents):**
-
-```bash
-export ONYX_SERVER_URL="https://your-onyx-server.com"  # default: https://cloud.onyx.app
-export ONYX_API_KEY="your-api-key"
-```
-
-Environment variables override the config file. If these are set, no config file is needed.
-
-| Variable | Required | Description |
-|----------|----------|-------------|
-| `ONYX_SERVER_URL` | No | Onyx server base URL (default: `https://cloud.onyx.app`) |
-| `ONYX_API_KEY` | Yes | API key for authentication |
-| `ONYX_PERSONA_ID` | No | Default agent/persona ID |
-
-If neither the config file nor environment variables are set, tell the user that `onyx-cli` needs to be configured and ask them to either:
-- Run `onyx-cli configure` interactively, or
-- Set `ONYX_SERVER_URL` and `ONYX_API_KEY` environment variables
-
-## Commands
-
-### Validate configuration
-
-```bash
-onyx-cli validate-config
-```
-
-Checks config file exists, API key is present, and tests the server connection. Use this before `ask` or `agents` to confirm the CLI is properly set up.
-
-### List available agents
-
-```bash
-onyx-cli agents
-```
-
-Prints a table of agent IDs, names, and descriptions. Use `--json` for structured output:
-
-```bash
-onyx-cli agents --json
-```
-
-Use agent IDs with `ask --agent-id` to query a specific agent.
-
-### Basic query (plain text output)
-
-```bash
-onyx-cli ask "What is our company's PTO policy?"
-```
-
-Streams the answer as plain text to stdout. Exit code 0 on success, non-zero on error.
-
-### JSON output (structured events)
-
-```bash
-onyx-cli ask --json "What authentication methods do we support?"
-```
-
-Outputs JSON-encoded parsed stream events (one object per line). Key event objects include message deltas, stop, errors, search-start, and citation payloads.
-
-Each line is a JSON object with this envelope:
-
-```json
-{"type": "<event_type>", "event": { ... }}
-```
-
-| Event Type | Description |
-|------------|-------------|
-| `message_delta` | Content token — concatenate all `content` fields for the full answer |
-| `stop` | Stream complete |
-| `error` | Error with `error` message field |
-| `search_tool_start` | Onyx started searching documents |
-| `citation_info` | Source citation — see shape below |
-
-`citation_info` event shape:
-
-```json
-{
-  "type": "citation_info",
-  "event": {
-    "citation_number": 1,
-    "document_id": "abc123def456",
-    "placement": {"turn_index": 0, "tab_index": 0, "sub_turn_index": null}
-  }
-}
-```
-
-`placement` is metadata about where in the conversation the citation appeared and can be ignored for most use cases.
-
-### Specify an agent
-
-```bash
-onyx-cli ask --agent-id 5 "Summarize our Q4 roadmap"
-```
-
-Uses a specific Onyx agent/persona instead of the default.
-
-### All flags
-
-| Flag | Type | Description |
-|------|------|-------------|
-| `--agent-id` | int | Agent ID to use (overrides default) |
-| `--json` | bool | Output raw NDJSON events instead of plain text |
-
-## Statelessness
-
-Each `onyx-cli ask` call creates an independent chat session. There is no built-in way to chain context across multiple `ask` invocations — every call starts fresh. If you need multi-turn conversation with memory, use the interactive TUI (`onyx-cli` or `onyx-cli chat`) instead.
-
-## When to Use
-
-Use `onyx-cli ask` when:
-
-- The user asks about company-specific information (policies, docs, processes)
-- You need to search internal knowledge bases or connected data sources
-- The user references Onyx, asks you to "search Onyx", or wants to query their documents
-- You need context from company wikis, Confluence, Google Drive, Slack, or other connected sources
-
-Do NOT use when:
-
-- The question is about general programming knowledge (use your own knowledge)
-- The user is asking about code in the current repository (use grep/read tools)
-- The user hasn't mentioned Onyx and the question doesn't require internal company data
-
-## Examples
-
-```bash
-# Simple question
-onyx-cli ask "What are the steps to deploy to production?"
-
-# Get structured output for parsing
-onyx-cli ask --json "List all active API integrations"
-
-# Use a specialized agent
-onyx-cli ask --agent-id 3 "What were the action items from last week's standup?"
-
-# Pipe the answer into another command
-onyx-cli ask "What is the database schema for users?" | head -20
-```

.github/CODEOWNERS

@@ -8,6 +8,3 @@
 # Agent context files
 /CLAUDE.md @Weves
 /AGENTS.md @Weves
-
-# Beta cherry-pick workflow owners
-/.github/workflows/post-merge-beta-cherry-pick.yml @justin-tahara @jmelahman

.github/actions/slack-notify/action.yml

@@ -1,14 +1,11 @@
-name: "Slack Notify"
-description: "Sends a Slack notification for workflow events"
+name: "Slack Notify on Failure"
+description: "Sends a Slack notification when a workflow fails"
 inputs:
   webhook-url:
     description: "Slack webhook URL (can also use SLACK_WEBHOOK_URL env var)"
     required: false
-  details:
-    description: "Additional message body content"
-    required: false
   failed-jobs:
-    description: "Deprecated alias for details"
+    description: "List of failed job names (newline-separated)"
     required: false
   title:
     description: "Title for the notification"
@@ -24,7 +21,6 @@ runs:
       shell: bash
       env:
         SLACK_WEBHOOK_URL: ${{ inputs.webhook-url }}
-        DETAILS: ${{ inputs.details }}
         FAILED_JOBS: ${{ inputs.failed-jobs }}
         TITLE: ${{ inputs.title }}
         REF_NAME: ${{ inputs.ref-name }}
@@ -48,18 +44,6 @@ runs:
           REF_NAME="$GITHUB_REF_NAME"
         fi
 
-        if [ -z "$DETAILS" ]; then
-          DETAILS="$FAILED_JOBS"
-        fi
-
-        normalize_multiline() {
-          printf '%s' "$1" | awk 'BEGIN { ORS=""; first=1 } { if (!first) printf "\\n"; printf "%s", $0; first=0 }'
-        }
-
-        DETAILS="$(normalize_multiline "$DETAILS")"
-        REF_NAME="$(normalize_multiline "$REF_NAME")"
-        TITLE="$(normalize_multiline "$TITLE")"
-
         # Escape JSON special characters
         escape_json() {
           local input="$1"
@@ -75,12 +59,12 @@ runs:
         }
 
         REF_NAME_ESC=$(escape_json "$REF_NAME")
-        DETAILS_ESC=$(escape_json "$DETAILS")
+        FAILED_JOBS_ESC=$(escape_json "$FAILED_JOBS")
         WORKFLOW_URL_ESC=$(escape_json "$WORKFLOW_URL")
         TITLE_ESC=$(escape_json "$TITLE")
 
         # Build JSON payload piece by piece
-        # Note: DETAILS_ESC already contains \n sequences that should remain as \n in JSON
+        # Note: FAILED_JOBS_ESC already contains \n sequences that should remain as \n in JSON
         PAYLOAD="{"
         PAYLOAD="${PAYLOAD}\"text\":\"${TITLE_ESC}\","
         PAYLOAD="${PAYLOAD}\"blocks\":[{"
@@ -95,10 +79,10 @@ runs:
         PAYLOAD="${PAYLOAD}{\"type\":\"mrkdwn\",\"text\":\"*Run ID:*\\n#${RUN_NUMBER}\"}"
         PAYLOAD="${PAYLOAD}]"
         PAYLOAD="${PAYLOAD}}"
-        if [ -n "$DETAILS" ]; then
+        if [ -n "$FAILED_JOBS" ]; then
           PAYLOAD="${PAYLOAD},{"
           PAYLOAD="${PAYLOAD}\"type\":\"section\","
-          PAYLOAD="${PAYLOAD}\"text\":{\"type\":\"mrkdwn\",\"text\":\"${DETAILS_ESC}\"}"
+          PAYLOAD="${PAYLOAD}\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Failed Jobs:*\\n${FAILED_JOBS_ESC}\"}"
           PAYLOAD="${PAYLOAD}}"
         fi
         PAYLOAD="${PAYLOAD},{"
@@ -115,3 +99,4 @@ runs:
         curl -X POST -H 'Content-type: application/json' \
           --data "$PAYLOAD" \
           "$SLACK_WEBHOOK_URL"
+

.github/workflows/deployment.yml

@@ -29,32 +29,20 @@ jobs:
       build-backend-craft: ${{ steps.check.outputs.build-backend-craft }}
       build-model-server: ${{ steps.check.outputs.build-model-server }}
       is-cloud-tag: ${{ steps.check.outputs.is-cloud-tag }}
+      is-stable: ${{ steps.check.outputs.is-stable }}
       is-beta: ${{ steps.check.outputs.is-beta }}
+      is-stable-standalone: ${{ steps.check.outputs.is-stable-standalone }}
       is-beta-standalone: ${{ steps.check.outputs.is-beta-standalone }}
-      is-latest: ${{ steps.check.outputs.is-latest }}
+      is-craft-latest: ${{ steps.check.outputs.is-craft-latest }}
       is-test-run: ${{ steps.check.outputs.is-test-run }}
       sanitized-tag: ${{ steps.check.outputs.sanitized-tag }}
       short-sha: ${{ steps.check.outputs.short-sha }}
     steps:
-      - name: Checkout (for git tags)
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-          fetch-depth: 0
-          fetch-tags: true
-
-      - name: Setup uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          version: "0.9.9"
-          enable-cache: false
-
       - name: Check which components to build and version info
         id: check
         env:
           EVENT_NAME: ${{ github.event_name }}
         run: |
-          set -eo pipefail
           TAG="${GITHUB_REF_NAME}"
           # Sanitize tag name by replacing slashes with hyphens (for Docker tag compatibility)
           SANITIZED_TAG=$(echo "$TAG" | tr '/' '-')
@@ -66,8 +54,9 @@ jobs:
           IS_VERSION_TAG=false
           IS_STABLE=false
           IS_BETA=false
+          IS_STABLE_STANDALONE=false
           IS_BETA_STANDALONE=false
-          IS_LATEST=false
+          IS_CRAFT_LATEST=false
           IS_PROD_TAG=false
           IS_TEST_RUN=false
           BUILD_DESKTOP=false
@@ -78,6 +67,9 @@ jobs:
           BUILD_MODEL_SERVER=true
 
           # Determine tag type based on pattern matching (do regex checks once)
+          if [[ "$TAG" == craft-* ]]; then
+            IS_CRAFT_LATEST=true
+          fi
           if [[ "$TAG" == *cloud* ]]; then
             IS_CLOUD=true
           fi
@@ -105,28 +97,20 @@ jobs:
             fi
           fi
 
+          # Craft-latest builds backend with Craft enabled
+          if [[ "$IS_CRAFT_LATEST" == "true" ]]; then
+            BUILD_BACKEND_CRAFT=true
+            BUILD_BACKEND=false
+          fi
+
           # Standalone version checks (for backend/model-server - version excluding cloud tags)
+          if [[ "$IS_STABLE" == "true" ]] && [[ "$IS_CLOUD" != "true" ]]; then
+            IS_STABLE_STANDALONE=true
+          fi
           if [[ "$IS_BETA" == "true" ]] && [[ "$IS_CLOUD" != "true" ]]; then
             IS_BETA_STANDALONE=true
           fi
 
-          # Determine if this tag should get the "latest" Docker tag.
-          # Only the highest semver stable tag (vX.Y.Z exactly) gets "latest".
-          if [[ "$IS_STABLE" == "true" ]]; then
-            HIGHEST_STABLE=$(uv run --no-sync --with onyx-devtools ods latest-stable-tag) || {
-              echo "::error::Failed to determine highest stable tag via 'ods latest-stable-tag'"
-              exit 1
-            }
-            if [[ "$TAG" == "$HIGHEST_STABLE" ]]; then
-              IS_LATEST=true
-            fi
-          fi
-
-          # Build craft-latest backend alongside the regular latest.
-          if [[ "$IS_LATEST" == "true" ]]; then
-            BUILD_BACKEND_CRAFT=true
-          fi
-
           # Determine if this is a production tag
           # Production tags are: version tags (v1.2.3*) or nightly tags
           if [[ "$IS_VERSION_TAG" == "true" ]] || [[ "$IS_NIGHTLY" == "true" ]]; then
@@ -145,9 +129,11 @@ jobs:
             echo "build-backend-craft=$BUILD_BACKEND_CRAFT"
             echo "build-model-server=$BUILD_MODEL_SERVER"
             echo "is-cloud-tag=$IS_CLOUD"
+            echo "is-stable=$IS_STABLE"
             echo "is-beta=$IS_BETA"
+            echo "is-stable-standalone=$IS_STABLE_STANDALONE"
             echo "is-beta-standalone=$IS_BETA_STANDALONE"
-            echo "is-latest=$IS_LATEST"
+            echo "is-craft-latest=$IS_CRAFT_LATEST"
             echo "is-test-run=$IS_TEST_RUN"
             echo "sanitized-tag=$SANITIZED_TAG"
             echo "short-sha=$SHORT_SHA"
@@ -165,7 +151,7 @@ jobs:
           fetch-depth: 0
 
       - name: Setup uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           version: "0.9.9"
           # NOTE: This isn't caching much and zizmor suggests this could be poisoned, so disable.
@@ -196,52 +182,8 @@ jobs:
           title: "🚨 Version Tag Check Failed"
           ref-name: ${{ github.ref_name }}
 
-  # Create GitHub release first, before desktop builds start.
-  # This ensures all desktop matrix jobs upload to the same release instead of
-  # racing to create duplicate releases.
-  create-release:
-    needs: determine-builds
-    if: needs.determine-builds.outputs.build-desktop == 'true'
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    permissions:
-      contents: write
-    outputs:
-      release-id: ${{ steps.create-release.outputs.id }}
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Determine release tag
-        id: release-tag
-        env:
-          IS_TEST_RUN: ${{ needs.determine-builds.outputs.is-test-run }}
-          SHORT_SHA: ${{ needs.determine-builds.outputs.short-sha }}
-        run: |
-          if [ "${IS_TEST_RUN}" == "true" ]; then
-            echo "tag=v0.0.0-dev+${SHORT_SHA}" >> "$GITHUB_OUTPUT"
-          else
-            echo "tag=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Create GitHub Release
-        id: create-release
-        uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # ratchet:softprops/action-gh-release@v2
-        with:
-          tag_name: ${{ steps.release-tag.outputs.tag }}
-          name: ${{ steps.release-tag.outputs.tag }}
-          body: "See the assets to download this version and install."
-          draft: true
-          prerelease: false
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
   build-desktop:
-    needs:
-      - determine-builds
-      - create-release
+    needs: determine-builds
     if: needs.determine-builds.outputs.build-desktop == 'true'
     permissions:
       id-token: write
@@ -266,12 +208,12 @@ jobs:
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6.0.2
         with:
-          # NOTE: persist-credentials is needed for tauri-action to upload assets to GitHub releases.
+          # NOTE: persist-credentials is needed for tauri-action to create GitHub releases.
           persist-credentials: true # zizmor: ignore[artipacked]
 
       - name: Configure AWS credentials
         if: startsWith(matrix.platform, 'macos-')
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -411,9 +353,11 @@ jobs:
           APPLE_SIGNING_IDENTITY: ${{ env.CERT_ID }}
           APPLE_TEAM_ID: ${{ env.APPLE_TEAM_ID }}
         with:
-          # Use the release created by the create-release job to avoid race conditions
-          # when multiple matrix jobs try to create/update the same release simultaneously
-          releaseId: ${{ needs.create-release.outputs.release-id }}
+          tagName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseBody: "See the assets to download this version and install."
+          releaseDraft: true
+          prerelease: false
           assetNamePattern: "[name]_[arch][ext]"
           args: ${{ matrix.args }}
 
@@ -440,7 +384,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -514,7 +458,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -583,7 +527,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -614,7 +558,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('web-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta == 'true' && 'beta' || '' }}
 
@@ -653,7 +597,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -735,7 +679,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -812,7 +756,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -879,7 +823,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -952,7 +896,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1020,7 +964,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1051,7 +995,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('backend-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable-standalone == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta-standalone == 'true' && 'beta' || '' }}
 
@@ -1090,7 +1034,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1163,7 +1107,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1232,7 +1176,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1302,7 +1246,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1382,7 +1326,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1456,7 +1400,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1487,7 +1431,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('model-server-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable-standalone == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta-standalone == 'true' && 'beta' || '' }}
 
@@ -1521,7 +1465,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1576,7 +1520,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1636,7 +1580,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1693,7 +1637,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2

.github/workflows/nightly-llm-provider-chat.yml

@@ -15,8 +15,6 @@ permissions:
 jobs:
   provider-chat-test:
     uses: ./.github/workflows/reusable-nightly-llm-provider-chat.yml
-    secrets:
-      AWS_OIDC_ROLE_ARN: ${{ secrets.AWS_OIDC_ROLE_ARN }}
     permissions:
       contents: read
       id-token: write

.github/workflows/post-merge-beta-cherry-pick.yml

@@ -1,112 +1,65 @@
 name: Post-Merge Beta Cherry-Pick
 
 on:
-  pull_request_target:
-    types:
-      - closed
+  push:
+    branches:
+      - main
 
-# SECURITY NOTE:
-# This workflow intentionally uses pull_request_target so post-merge automation can
-# use base-repo credentials. Do not checkout PR head refs in this workflow
-# (e.g. github.event.pull_request.head.sha). Only trusted base refs are allowed.
 permissions:
-  contents: read
+  contents: write
+  pull-requests: write
 
 jobs:
-  resolve-cherry-pick-request:
-    if: >-
-      github.event.pull_request.merged == true
-      && github.event.pull_request.base.ref == 'main'
-      && github.event.pull_request.head.repo.full_name == github.repository
+  cherry-pick-to-latest-release:
     outputs:
       should_cherrypick: ${{ steps.gate.outputs.should_cherrypick }}
       pr_number: ${{ steps.gate.outputs.pr_number }}
-      merge_commit_sha: ${{ steps.gate.outputs.merge_commit_sha }}
-      merged_by: ${{ steps.gate.outputs.merged_by }}
-      gate_error: ${{ steps.gate.outputs.gate_error }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - name: Resolve merged PR and checkbox state
-        id: gate
-        env:
-          GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
-          # SECURITY: keep PR body in env/plain-text handling; avoid directly
-          # inlining github.event.pull_request.body into shell commands.
-          PR_BODY: ${{ github.event.pull_request.body }}
-          MERGE_COMMIT_SHA: ${{ github.event.pull_request.merge_commit_sha }}
-          MERGED_BY: ${{ github.event.pull_request.merged_by.login }}
-          # Explicit merger allowlist used because pull_request_target runs with
-          # the default GITHUB_TOKEN, which cannot reliably read org/team
-          # membership for this repository context.
-          ALLOWED_MERGERS: |
-            acaprau
-            bo-onyx
-            danelegend
-            duo-onyx
-            evan-onyx
-            jessicasingh7
-            jmelahman
-            joachim-danswer
-            justin-tahara
-            nmgarza5
-            raunakab
-            rohoswagger
-            subash-mohan
-            trial2onyx
-            wenxi-onyx
-            weves
-            yuhongsun96
-        run: |
-          echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
-          echo "merged_by=${MERGED_BY}" >> "$GITHUB_OUTPUT"
-
-          if ! echo "${PR_BODY}" | grep -qiE "\\[x\\][[:space:]]*(\\[[^]]+\\][[:space:]]*)?Please cherry-pick this PR to the latest release version"; then
-            echo "should_cherrypick=false" >> "$GITHUB_OUTPUT"
-            echo "Cherry-pick checkbox not checked for PR #${PR_NUMBER}. Skipping."
-            exit 0
-          fi
-
-          # Keep should_cherrypick output before any possible exit 1 below so
-          # notify-slack can still gate on this output even if this job fails.
-          echo "should_cherrypick=true" >> "$GITHUB_OUTPUT"
-          echo "Cherry-pick checkbox checked for PR #${PR_NUMBER}."
-
-          if [ -z "${MERGE_COMMIT_SHA}" ] || [ "${MERGE_COMMIT_SHA}" = "null" ]; then
-            echo "gate_error=missing-merge-commit-sha" >> "$GITHUB_OUTPUT"
-            echo "::error::PR #${PR_NUMBER} requested cherry-pick, but merge_commit_sha is missing."
-            exit 1
-          fi
-
-          echo "merge_commit_sha=${MERGE_COMMIT_SHA}" >> "$GITHUB_OUTPUT"
-
-          normalized_merged_by="$(printf '%s' "${MERGED_BY}" | tr '[:upper:]' '[:lower:]')"
-          normalized_allowed_mergers="$(printf '%s\n' "${ALLOWED_MERGERS}" | tr '[:upper:]' '[:lower:]')"
-          if ! printf '%s\n' "${normalized_allowed_mergers}" | grep -Fxq "${normalized_merged_by}"; then
-            echo "gate_error=not-allowed-merger" >> "$GITHUB_OUTPUT"
-            echo "::error::${MERGED_BY} is not in the explicit cherry-pick merger allowlist. Failing cherry-pick gate."
-            exit 1
-          fi
-
-          exit 0
-
-  cherry-pick-to-latest-release:
-    needs:
-      - resolve-cherry-pick-request
-    if: needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && needs.resolve-cherry-pick-request.result == 'success'
-    permissions:
-      contents: write
-      pull-requests: write
-    outputs:
-      cherry_pick_pr_url: ${{ steps.run_cherry_pick.outputs.pr_url }}
       cherry_pick_reason: ${{ steps.run_cherry_pick.outputs.reason }}
       cherry_pick_details: ${{ steps.run_cherry_pick.outputs.details }}
     runs-on: ubuntu-latest
     timeout-minutes: 45
     steps:
+      - name: Resolve merged PR and checkbox state
+        id: gate
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          # For the commit that triggered this workflow (HEAD on main), fetch all
+          # associated PRs and keep only the PR that was actually merged into main
+          # with this exact merge commit SHA.
+          pr_numbers="$(gh api "repos/${GITHUB_REPOSITORY}/commits/${GITHUB_SHA}/pulls" | jq -r --arg sha "${GITHUB_SHA}" '.[] | select(.merged_at != null and .base.ref == "main" and .merge_commit_sha == $sha) | .number')"
+          match_count="$(printf '%s\n' "$pr_numbers" | sed '/^[[:space:]]*$/d' | wc -l | tr -d ' ')"
+          pr_number="$(printf '%s\n' "$pr_numbers" | sed '/^[[:space:]]*$/d' | head -n 1)"
+
+          if [ "${match_count}" -gt 1 ]; then
+            echo "::warning::Multiple merged PRs matched commit ${GITHUB_SHA}. Using PR #${pr_number}."
+          fi
+
+          if [ -z "$pr_number" ]; then
+            echo "No merged PR associated with commit ${GITHUB_SHA}; skipping."
+            echo "should_cherrypick=false" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          # Read the PR once so we can gate behavior and infer preferred actor.
+          pr_json="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${pr_number}")"
+          pr_body="$(printf '%s' "$pr_json" | jq -r '.body // ""')"
+          merged_by="$(printf '%s' "$pr_json" | jq -r '.merged_by.login // ""')"
+
+          echo "pr_number=$pr_number" >> "$GITHUB_OUTPUT"
+          echo "merged_by=$merged_by" >> "$GITHUB_OUTPUT"
+
+          if echo "$pr_body" | grep -qiE "\\[x\\][[:space:]]*(\\[[^]]+\\][[:space:]]*)?Please cherry-pick this PR to the latest release version"; then
+            echo "should_cherrypick=true" >> "$GITHUB_OUTPUT"
+            echo "Cherry-pick checkbox checked for PR #${pr_number}."
+            exit 0
+          fi
+
+          echo "should_cherrypick=false" >> "$GITHUB_OUTPUT"
+          echo "Cherry-pick checkbox not checked for PR #${pr_number}. Skipping."
+
       - name: Checkout repository
-        # SECURITY: keep checkout pinned to trusted base branch; do not switch to PR head refs.
+        if: steps.gate.outputs.should_cherrypick == 'true'
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
         with:
           fetch-depth: 0
@@ -114,44 +67,34 @@ jobs:
           ref: main
 
       - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        if: steps.gate.outputs.should_cherrypick == 'true'
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"
 
       - name: Configure git identity
+        if: steps.gate.outputs.should_cherrypick == 'true'
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "github-actions[bot]@users.noreply.github.com"
 
       - name: Create cherry-pick PR to latest release
         id: run_cherry_pick
+        if: steps.gate.outputs.should_cherrypick == 'true'
+        continue-on-error: true
         env:
           GH_TOKEN: ${{ github.token }}
           GITHUB_TOKEN: ${{ github.token }}
-          CHERRY_PICK_ASSIGNEE: ${{ needs.resolve-cherry-pick-request.outputs.merged_by }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
+          CHERRY_PICK_ASSIGNEE: ${{ steps.gate.outputs.merged_by }}
         run: |
+          set -o pipefail
           output_file="$(mktemp)"
-          set +e
-          uv run --no-sync --with onyx-devtools ods cherry-pick "${MERGE_COMMIT_SHA}" --yes --no-verify 2>&1 | tee "$output_file"
-          pipe_statuses=("${PIPESTATUS[@]}")
-          exit_code="${pipe_statuses[0]}"
-          tee_exit="${pipe_statuses[1]:-0}"
-          set -e
-          if [ "${tee_exit}" -ne 0 ]; then
-            echo "status=failure" >> "$GITHUB_OUTPUT"
-            echo "reason=output-capture-failed" >> "$GITHUB_OUTPUT"
-            echo "::error::tee failed to capture cherry-pick output (exit ${tee_exit}); cannot classify result."
-            exit 1
-          fi
+          uv run --no-sync --with onyx-devtools ods cherry-pick "${GITHUB_SHA}" --yes --no-verify 2>&1 | tee "$output_file"
+          exit_code="${PIPESTATUS[0]}"
 
           if [ "${exit_code}" -eq 0 ]; then
-            pr_url="$(sed -n 's/^.*PR created successfully: \(https:\/\/github\.com\/[^[:space:]]\+\/pull\/[0-9]\+\).*$/\1/p' "$output_file" | tail -n 1)"
             echo "status=success" >> "$GITHUB_OUTPUT"
-            if [ -n "${pr_url}" ]; then
-              echo "pr_url=${pr_url}" >> "$GITHUB_OUTPUT"
-            fi
             exit 0
           fi
 
@@ -170,66 +113,17 @@ jobs:
           } >> "$GITHUB_OUTPUT"
 
       - name: Mark workflow as failed if cherry-pick failed
-        if: steps.run_cherry_pick.outputs.status == 'failure'
+        if: steps.gate.outputs.should_cherrypick == 'true' && steps.run_cherry_pick.outputs.status == 'failure'
         env:
           CHERRY_PICK_REASON: ${{ steps.run_cherry_pick.outputs.reason }}
         run: |
           echo "::error::Automated cherry-pick failed (${CHERRY_PICK_REASON})."
           exit 1
 
-  notify-slack-on-cherry-pick-success:
-    needs:
-      - resolve-cherry-pick-request
-      - cherry-pick-to-latest-release
-    if: needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && needs.resolve-cherry-pick-request.result == 'success' && needs.cherry-pick-to-latest-release.result == 'success'
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Fail if Slack webhook secret is missing
-        env:
-          CHERRY_PICK_PRS_WEBHOOK: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-        run: |
-          if [ -z "${CHERRY_PICK_PRS_WEBHOOK}" ]; then
-            echo "::error::CHERRY_PICK_PRS_WEBHOOK is not configured."
-            exit 1
-          fi
-
-      - name: Build cherry-pick success summary
-        id: success-summary
-        env:
-          SOURCE_PR_NUMBER: ${{ needs.resolve-cherry-pick-request.outputs.pr_number }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
-          CHERRY_PICK_PR_URL: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_pr_url }}
-        run: |
-          source_pr_url="https://github.com/${GITHUB_REPOSITORY}/pull/${SOURCE_PR_NUMBER}"
-          details="*Cherry-pick PR opened successfully.*\\n• source PR: ${source_pr_url}"
-          if [ -n "${CHERRY_PICK_PR_URL}" ]; then
-            details="${details}\\n• cherry-pick PR: ${CHERRY_PICK_PR_URL}"
-          fi
-          if [ -n "${MERGE_COMMIT_SHA}" ]; then
-            details="${details}\\n• merge SHA: ${MERGE_COMMIT_SHA}"
-          fi
-
-          echo "details=${details}" >> "$GITHUB_OUTPUT"
-
-      - name: Notify #cherry-pick-prs about cherry-pick success
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-          details: ${{ steps.success-summary.outputs.details }}
-          title: "✅ Automated Cherry-Pick PR Opened"
-          ref-name: ${{ github.event.pull_request.base.ref }}
-
   notify-slack-on-cherry-pick-failure:
     needs:
-      - resolve-cherry-pick-request
       - cherry-pick-to-latest-release
-    if: always() && needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && (needs.resolve-cherry-pick-request.result == 'failure' || needs.cherry-pick-to-latest-release.result == 'failure')
+    if: always() && needs.cherry-pick-to-latest-release.outputs.should_cherrypick == 'true' && needs.cherry-pick-to-latest-release.result != 'success'
     runs-on: ubuntu-slim
     timeout-minutes: 10
     steps:
@@ -238,47 +132,22 @@ jobs:
         with:
           persist-credentials: false
 
-      - name: Fail if Slack webhook secret is missing
-        env:
-          CHERRY_PICK_PRS_WEBHOOK: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-        run: |
-          if [ -z "${CHERRY_PICK_PRS_WEBHOOK}" ]; then
-            echo "::error::CHERRY_PICK_PRS_WEBHOOK is not configured."
-            exit 1
-          fi
-
       - name: Build cherry-pick failure summary
         id: failure-summary
         env:
-          SOURCE_PR_NUMBER: ${{ needs.resolve-cherry-pick-request.outputs.pr_number }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
-          GATE_ERROR: ${{ needs.resolve-cherry-pick-request.outputs.gate_error }}
+          SOURCE_PR_NUMBER: ${{ needs.cherry-pick-to-latest-release.outputs.pr_number }}
           CHERRY_PICK_REASON: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_reason }}
           CHERRY_PICK_DETAILS: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_details }}
         run: |
           source_pr_url="https://github.com/${GITHUB_REPOSITORY}/pull/${SOURCE_PR_NUMBER}"
 
           reason_text="cherry-pick command failed"
-          if [ "${GATE_ERROR}" = "missing-merge-commit-sha" ]; then
-            reason_text="requested cherry-pick but merge commit SHA was missing"
-          elif [ "${GATE_ERROR}" = "not-allowed-merger" ]; then
-            reason_text="merger is not in the explicit cherry-pick allowlist"
-          elif [ "${CHERRY_PICK_REASON}" = "output-capture-failed" ]; then
-            reason_text="failed to capture cherry-pick output for classification"
-          elif [ "${CHERRY_PICK_REASON}" = "merge-conflict" ]; then
+          if [ "${CHERRY_PICK_REASON}" = "merge-conflict" ]; then
             reason_text="merge conflict during cherry-pick"
           fi
 
           details_excerpt="$(printf '%s' "${CHERRY_PICK_DETAILS}" | tail -n 8 | tr '\n' ' ' | sed "s/[[:space:]]\\+/ /g" | sed "s/\"/'/g" | cut -c1-350)"
-          if [ -n "${GATE_ERROR}" ]; then
-            failed_job_label="resolve-cherry-pick-request"
-          else
-            failed_job_label="cherry-pick-to-latest-release"
-          fi
-          failed_jobs="• ${failed_job_label}\\n• source PR: ${source_pr_url}\\n• reason: ${reason_text}"
-          if [ -n "${MERGE_COMMIT_SHA}" ]; then
-            failed_jobs="${failed_jobs}\\n• merge SHA: ${MERGE_COMMIT_SHA}"
-          fi
+          failed_jobs="• cherry-pick-to-latest-release\\n• source PR: ${source_pr_url}\\n• reason: ${reason_text}"
           if [ -n "${details_excerpt}" ]; then
             failed_jobs="${failed_jobs}\\n• excerpt: ${details_excerpt}"
           fi
@@ -289,6 +158,6 @@ jobs:
         uses: ./.github/actions/slack-notify
         with:
           webhook-url: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-          details: ${{ steps.failure-summary.outputs.jobs }}
+          failed-jobs: ${{ steps.failure-summary.outputs.jobs }}
           title: "🚨 Automated Cherry-Pick Failed"
-          ref-name: ${{ github.event.pull_request.base.ref }}
+          ref-name: ${{ github.ref_name }}

.github/workflows/pr-desktop-build.yml

@@ -57,7 +57,7 @@ jobs:
           cache-dependency-path: ./desktop/package-lock.json
 
       - name: Setup Rust
-        uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9
+        uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7
         with:
           toolchain: stable
           targets: ${{ matrix.target }}

.github/workflows/pr-golang-tests.yml

@@ -1,56 +0,0 @@
-name: Golang Tests
-concurrency:
-  group: Golang-Tests-${{ github.workflow }}-${{ github.head_ref || github.event.workflow_run.head_branch || github.run_id }}
-  cancel-in-progress: true
-
-on:
-  merge_group:
-  pull_request:
-    branches:
-      - main
-      - "release/**"
-  push:
-    tags:
-      - "v*.*.*"
-
-permissions: {}
-
-env:
-  GO_VERSION: "1.26"
-
-jobs:
-  detect-modules:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      modules: ${{ steps.set-modules.outputs.modules }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
-        with:
-          persist-credentials: false
-      - id: set-modules
-        run: echo "modules=$(find . -name 'go.mod' -exec dirname {} \; | jq -Rc '[.,inputs]')" >> "$GITHUB_OUTPUT"
-
-  golang:
-    needs: detect-modules
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        modules: ${{ fromJSON(needs.detect-modules.outputs.modules) }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # zizmor: ignore[cache-poisoning]
-        with:
-          go-version: ${{ env.GO_VERSION }}
-          cache-dependency-path: "**/go.sum"
-
-      - run: go mod tidy
-        working-directory: ${{ matrix.modules }}
-      - run: git diff --exit-code go.mod go.sum
-        working-directory: ${{ matrix.modules }}
-
-      - run: go test ./...
-        working-directory: ${{ matrix.modules }}

.github/workflows/pr-helm-chart-testing.yml

@@ -71,7 +71,7 @@ jobs:
 
       - name: Create kind cluster
         if: steps.list-changed.outputs.changed == 'true'
-        uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # ratchet:helm/kind-action@v1.14.0
+        uses: helm/kind-action@92086f6be054225fa813e0a4b13787fc9088faab # ratchet:helm/kind-action@v1.13.0
 
       - name: Pre-install cluster status check
         if: steps.list-changed.outputs.changed == 'true'
@@ -133,7 +133,7 @@ jobs:
           echo "=== Validating chart dependencies ==="
           cd deployment/helm/charts/onyx
           helm dependency update
-          helm lint . --set auth.userauth.values.user_auth_secret=placeholder
+          helm lint .
 
       - name: Run chart-testing (install) with enhanced monitoring
         timeout-minutes: 25
@@ -194,7 +194,6 @@ jobs:
               --set=vespa.enabled=false \
               --set=opensearch.enabled=true \
               --set=auth.opensearch.enabled=true \
-              --set=auth.userauth.values.user_auth_secret=test-secret \
               --set=slackbot.enabled=false \
               --set=postgresql.enabled=true \
               --set=postgresql.cluster.storage.storageClass=standard \
@@ -231,10 +230,6 @@ jobs:
         if: steps.list-changed.outputs.changed == 'true'
         run: |
           echo "=== Post-install verification ==="
-          if ! kubectl cluster-info >/dev/null 2>&1; then
-            echo "ERROR: Kubernetes cluster is not reachable after install"
-            exit 1
-          fi
           kubectl get pods --all-namespaces
           kubectl get services --all-namespaces
           # Only show issues if they exist
@@ -244,10 +239,6 @@ jobs:
         if: failure() && steps.list-changed.outputs.changed == 'true'
         run: |
           echo "=== Cleanup on failure ==="
-          if ! kubectl cluster-info >/dev/null 2>&1; then
-            echo "Skipping failure cleanup: Kubernetes cluster is not reachable"
-            exit 0
-          fi
           echo "=== Final cluster state ==="
           kubectl get pods --all-namespaces
           kubectl get events --all-namespaces --sort-by=.lastTimestamp | tail -10

.github/workflows/pr-integration-tests.yml

@@ -316,7 +316,6 @@ jobs:
           # Base config shared by both editions
           cat <<EOF > deployment/docker_compose/.env
           COMPOSE_PROFILES=s3-filestore
-          OPENSEARCH_FOR_ONYX_ENABLED=false
           AUTH_TYPE=basic
           POSTGRES_POOL_PRE_PING=true
           POSTGRES_USE_NULL_POOL=true
@@ -336,6 +335,7 @@ jobs:
           # TODO(Nik): https://linear.app/onyx-app/issue/ENG-1/update-test-infra-to-use-test-license
           LICENSE_ENFORCEMENT_ENABLED=false
           CHECK_TTL_MANAGEMENT_TASK_FREQUENCY_IN_HOURS=0.001
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=false
           EOF
           fi
 
@@ -419,7 +419,6 @@ jobs:
               -e POSTGRES_POOL_PRE_PING=true \
               -e POSTGRES_USE_NULL_POOL=true \
               -e VESPA_HOST=index \
-              -e ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false \
               -e REDIS_HOST=cache \
               -e API_SERVER_HOST=api_server \
               -e OPENAI_API_KEY=${OPENAI_API_KEY} \
@@ -472,13 +471,13 @@ jobs:
           path: ${{ github.workspace }}/docker-compose.log
       # ------------------------------------------------------------
 
-  onyx-lite-tests:
+  no-vectordb-tests:
     needs: [build-backend-image, build-integration-image]
     runs-on:
       [
         runs-on,
         runner=4cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-onyx-lite-tests",
+        "run-id=${{ github.run_id }}-no-vectordb-tests",
         "extras=ecr-cache",
       ]
     timeout-minutes: 45
@@ -496,12 +495,13 @@ jobs:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_TOKEN }}
 
-      - name: Create .env file for Onyx Lite Docker Compose
+      - name: Create .env file for no-vectordb Docker Compose
         env:
           ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
           RUN_ID: ${{ github.run_id }}
         run: |
           cat <<EOF > deployment/docker_compose/.env
+          COMPOSE_PROFILES=s3-filestore
           ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
           LICENSE_ENFORCEMENT_ENABLED=false
           AUTH_TYPE=basic
@@ -509,23 +509,28 @@ jobs:
           POSTGRES_USE_NULL_POOL=true
           REQUIRE_EMAIL_VERIFICATION=false
           DISABLE_TELEMETRY=true
+          DISABLE_VECTOR_DB=true
           ONYX_BACKEND_IMAGE=${ECR_CACHE}:integration-test-backend-test-${RUN_ID}
           INTEGRATION_TESTS_MODE=true
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=true
           EOF
 
-      # Start only the services needed for Onyx Lite (Postgres + API server)
-      - name: Start Docker containers (onyx-lite)
+      # Start only the services needed for no-vectordb mode (no Vespa, no model servers)
+      - name: Start Docker containers (no-vectordb)
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up \
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml up \
             relational_db \
+            cache \
+            minio \
             api_server \
+            background \
             -d
-        id: start_docker_onyx_lite
+        id: start_docker_no_vectordb
 
       - name: Wait for services to be ready
         run: |
-          echo "Starting wait-for-service script (onyx-lite)..."
+          echo "Starting wait-for-service script (no-vectordb)..."
           start_time=$(date +%s)
           timeout=300
           while true; do
@@ -547,14 +552,14 @@ jobs:
             sleep 5
           done
 
-      - name: Run Onyx Lite Integration Tests
+      - name: Run No-VectorDB Integration Tests
         uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # ratchet:nick-fields/retry@v3
         with:
           timeout_minutes: 20
           max_attempts: 3
           retry_wait_seconds: 10
           command: |
-            echo "Running onyx-lite integration tests..."
+            echo "Running no-vectordb integration tests..."
             docker run --rm --network onyx_default \
               --name test-runner \
               -e POSTGRES_HOST=relational_db \
@@ -565,38 +570,39 @@ jobs:
               -e DB_READONLY_PASSWORD=password \
               -e POSTGRES_POOL_PRE_PING=true \
               -e POSTGRES_USE_NULL_POOL=true \
+              -e REDIS_HOST=cache \
               -e API_SERVER_HOST=api_server \
               -e OPENAI_API_KEY=${OPENAI_API_KEY} \
               -e TEST_WEB_HOSTNAME=test-runner \
               ${{ env.RUNS_ON_ECR_CACHE }}:integration-test-${{ github.run_id }} \
               /app/tests/integration/tests/no_vectordb
 
-      - name: Dump API server logs (onyx-lite)
+      - name: Dump API server logs (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_onyx_lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_no_vectordb.log || true
 
-      - name: Dump all-container logs (onyx-lite)
+      - name: Dump all-container logs (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color > $GITHUB_WORKSPACE/docker-compose-onyx-lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color > $GITHUB_WORKSPACE/docker-compose-no-vectordb.log || true
 
-      - name: Upload logs (onyx-lite)
+      - name: Upload logs (no-vectordb)
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
         with:
-          name: docker-all-logs-onyx-lite
-          path: ${{ github.workspace }}/docker-compose-onyx-lite.log
+          name: docker-all-logs-no-vectordb
+          path: ${{ github.workspace }}/docker-compose-no-vectordb.log
 
-      - name: Stop Docker containers (onyx-lite)
+      - name: Stop Docker containers (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml down -v
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml down -v
 
   multitenant-tests:
     needs:
@@ -639,7 +645,6 @@ jobs:
           ONYX_BACKEND_IMAGE=${ECR_CACHE}:integration-test-backend-test-${RUN_ID} \
           ONYX_MODEL_SERVER_IMAGE=${ECR_CACHE}:integration-test-model-server-test-${RUN_ID} \
           DEV_MODE=true \
-          OPENSEARCH_FOR_ONYX_ENABLED=false \
           docker compose -f docker-compose.multitenant-dev.yml up \
             relational_db \
             index \
@@ -694,7 +699,6 @@ jobs:
             -e POSTGRES_DB=postgres \
             -e POSTGRES_USE_NULL_POOL=true \
             -e VESPA_HOST=index \
-            -e ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false \
             -e REDIS_HOST=cache \
             -e API_SERVER_HOST=api_server \
             -e OPENAI_API_KEY=${OPENAI_API_KEY} \
@@ -740,7 +744,7 @@ jobs:
     # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
     runs-on: ubuntu-slim
     timeout-minutes: 45
-    needs: [integration-tests, onyx-lite-tests, multitenant-tests]
+    needs: [integration-tests, no-vectordb-tests, multitenant-tests]
     if: ${{ always() }}
     steps:
       - name: Check job status

.github/workflows/pr-jest-tests.yml

@@ -31,7 +31,7 @@ jobs:
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
         with:
           node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning] test-only workflow; no deploy artifacts
+          cache: "npm"
           cache-dependency-path: ./web/package-lock.json
 
       - name: Install node dependencies

.github/workflows/pr-playwright-tests.yml

@@ -12,9 +12,6 @@ on:
   push:
     tags:
       - "v*.*.*"
-    # TODO: Remove this if we enable merge-queues for release branches.
-    branches:
-      - "release/**"
 
 permissions:
   contents: read
@@ -271,11 +268,10 @@ jobs:
           persist-credentials: false
 
       - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
         with:
           node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
+          cache: "npm"
           cache-dependency-path: ./web/package-lock.json
 
       - name: Install node dependencies
@@ -283,7 +279,6 @@ jobs:
         run: npm ci
 
       - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
         uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
         with:
           path: ~/.cache/ms-playwright
@@ -464,14 +459,14 @@ jobs:
       # --- Visual Regression Diff ---
       - name: Configure AWS credentials
         if: always()
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
 
       - name: Install the latest version of uv
         if: always()
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"
@@ -595,108 +590,6 @@ jobs:
           name: docker-logs-${{ matrix.project }}-${{ github.run_id }}
           path: ${{ github.workspace }}/docker-compose.log
 
-  playwright-tests-lite:
-    needs: [build-web-image, build-backend-image]
-    name: Playwright Tests (lite)
-    runs-on:
-      - runs-on
-      - runner=4cpu-linux-arm64
-      - "run-id=${{ github.run_id }}-playwright-tests-lite"
-      - "extras=ecr-cache"
-    timeout-minutes: 30
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
-        with:
-          node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
-          cache-dependency-path: ./web/package-lock.json
-
-      - name: Install node dependencies
-        working-directory: ./web
-        run: npm ci
-
-      - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
-        with:
-          path: ~/.cache/ms-playwright
-          key: ${{ runner.os }}-playwright-npm-${{ hashFiles('web/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-playwright-npm-
-
-      - name: Install playwright browsers
-        working-directory: ./web
-        run: npx playwright install --with-deps
-
-      - name: Create .env file for Docker Compose
-        env:
-          OPENAI_API_KEY_VALUE: ${{ env.OPENAI_API_KEY }}
-          ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
-          RUN_ID: ${{ github.run_id }}
-        run: |
-          cat <<EOF > deployment/docker_compose/.env
-          ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
-          LICENSE_ENFORCEMENT_ENABLED=false
-          AUTH_TYPE=basic
-          INTEGRATION_TESTS_MODE=true
-          GEN_AI_API_KEY=${OPENAI_API_KEY_VALUE}
-          MOCK_LLM_RESPONSE=true
-          REQUIRE_EMAIL_VERIFICATION=false
-          DISABLE_TELEMETRY=true
-          ONYX_BACKEND_IMAGE=${ECR_CACHE}:playwright-test-backend-${RUN_ID}
-          ONYX_WEB_SERVER_IMAGE=${ECR_CACHE}:playwright-test-web-${RUN_ID}
-          EOF
-
-      # needed for pulling external images otherwise, we hit the "Unauthenticated users" limit
-      # https://docs.docker.com/docker-hub/usage/
-      - name: Login to Docker Hub
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_TOKEN }}
-
-      - name: Start Docker containers (lite)
-        run: |
-          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up -d
-        id: start_docker
-
-      - name: Run Playwright tests (lite)
-        working-directory: ./web
-        run: npx playwright test --project lite
-
-      - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        if: always()
-        with:
-          name: playwright-test-results-lite-${{ github.run_id }}
-          path: ./web/output/playwright/
-          retention-days: 30
-
-      - name: Save Docker logs
-        if: success() || failure()
-        env:
-          WORKSPACE: ${{ github.workspace }}
-        run: |
-          cd deployment/docker_compose
-          docker compose logs > docker-compose.log
-          mv docker-compose.log ${WORKSPACE}/docker-compose.log
-
-      - name: Upload logs
-        if: success() || failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        with:
-          name: docker-logs-lite-${{ github.run_id }}
-          path: ${{ github.workspace }}/docker-compose.log
-
   # Post a single combined visual regression comment after all matrix jobs finish
   visual-regression-comment:
     needs: [playwright-tests]
@@ -710,7 +603,7 @@ jobs:
       pull-requests: write
     steps:
       - name: Download visual diff summaries
-        uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131
         with:
           pattern: screenshot-diff-summary-*
           path: summaries/
@@ -793,7 +686,7 @@ jobs:
     # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
     runs-on: ubuntu-slim
     timeout-minutes: 45
-    needs: [playwright-tests, playwright-tests-lite]
+    needs: [playwright-tests]
     if: ${{ always() }}
     steps:
       - name: Check job status

.github/workflows/pr-quality-checks.yml

@@ -28,7 +28,7 @@ jobs:
         with:
           python-version: "3.11"
       - name: Setup Terraform
-        uses: hashicorp/setup-terraform@5e8dbf3c6d9deaf4193ca7a8fb23f2ac83bb6c85 # ratchet:hashicorp/setup-terraform@v4.0.0
+        uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # ratchet:hashicorp/setup-terraform@v3
       - name: Setup node
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v6
         with: # zizmor: ignore[cache-poisoning]
@@ -38,9 +38,9 @@ jobs:
       - name: Install node dependencies
         working-directory: ./web
         run: npm ci
-      - uses: j178/prek-action@0bb87d7f00b0c99306c8bcb8b8beba1eb581c037 # ratchet:j178/prek-action@v1
+      - uses: j178/prek-action@9d6a3097e0c1865ecce00cfb89fe80f2ee91b547 # ratchet:j178/prek-action@v1
         with:
-          prek-version: '0.3.4'
+          prek-version: '0.2.21'
           extra-args: ${{ github.event_name == 'pull_request' && format('--from-ref {0} --to-ref {1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha) || github.event_name == 'merge_group' && format('--from-ref {0} --to-ref {1}', github.event.merge_group.base_sha, github.event.merge_group.head_sha) || github.ref_name == 'main' && '--all-files' || '' }}
       - name: Check Actions
         uses: giner/check-actions@28d366c7cbbe235f9624a88aa31a628167eee28c # ratchet:giner/check-actions@v1.0.1

.github/workflows/release-cli.yml

@@ -1,214 +0,0 @@
-name: Release CLI
-
-on:
-  push:
-    tags:
-      - "cli/v*.*.*"
-
-jobs:
-  pypi:
-    runs-on: ubuntu-latest
-    environment:
-      name: release-cli
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        os-arch:
-          - { goos: "linux", goarch: "amd64" }
-          - { goos: "linux", goarch: "arm64" }
-          - { goos: "windows", goarch: "amd64" }
-          - { goos: "windows", goarch: "arm64" }
-          - { goos: "darwin", goarch: "amd64" }
-          - { goos: "darwin", goarch: "arm64" }
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          enable-cache: false
-          version: "0.9.9"
-      - run: |
-          GOOS="${{ matrix.os-arch.goos }}" \
-          GOARCH="${{ matrix.os-arch.goarch }}" \
-          uv build --wheel
-        working-directory: cli
-      - run: uv publish
-        working-directory: cli
-
-  docker-amd64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-amd64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push AMD64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/amd64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  docker-arm64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-arm64
-      - run-id=${{ github.run_id }}-cli-arm64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push ARM64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/arm64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  merge-docker:
-    needs:
-      - docker-amd64
-      - docker-arm64
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-merge
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Create and push manifest
-        env:
-          AMD64_DIGEST: ${{ needs.docker-amd64.outputs.digest }}
-          ARM64_DIGEST: ${{ needs.docker-arm64.outputs.digest }}
-          TAG: ${{ github.ref_name }}
-        run: |
-          SANITIZED_TAG="${TAG#cli/}"
-          IMAGES=(
-            "${REGISTRY_IMAGE}@${AMD64_DIGEST}"
-            "${REGISTRY_IMAGE}@${ARM64_DIGEST}"
-          )
-
-          if [[ "$TAG" =~ ^cli/v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              -t "${REGISTRY_IMAGE}:latest" \
-              "${IMAGES[@]}"
-          else
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              "${IMAGES[@]}"
-          fi

.github/workflows/release-devtools.yml

@@ -22,11 +22,13 @@ jobs:
           - { goos: "windows", goarch: "arm64" }
           - { goos: "darwin", goarch: "amd64" }
           - { goos: "darwin", goarch: "arm64" }
+          - { goos: "", goarch: "" }
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
         with:
           persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+          fetch-depth: 0
+      - uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"

.github/workflows/reusable-nightly-llm-provider-chat.yml

@@ -48,10 +48,6 @@ on:
         required: false
         default: true
         type: boolean
-    secrets:
-      AWS_OIDC_ROLE_ARN:
-        description: "AWS role ARN for OIDC auth"
-        required: true
 
 permissions:
   contents: read
@@ -77,7 +73,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -120,7 +116,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -162,7 +158,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -268,7 +264,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2

.github/workflows/sandbox-deployment.yml

@@ -110,7 +110,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -180,7 +180,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -244,7 +244,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2

.github/workflows/storybook-deploy.yml

@@ -1,69 +0,0 @@
-name: Storybook Deploy
-env:
-  VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
-  VERCEL_PROJECT_ID: prj_sG49mVsA25UsxIPhN2pmBJlikJZM
-  VERCEL_CLI: vercel@50.14.1
-  VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
-
-concurrency:
-  group: storybook-deploy-production
-  cancel-in-progress: true
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
-    paths:
-      - "web/lib/opal/**"
-      - "web/src/refresh-components/**"
-      - "web/.storybook/**"
-      - "web/package.json"
-      - "web/package-lock.json"
-permissions:
-  contents: read
-jobs:
-  Deploy-Storybook:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v4
-        with:
-          persist-credentials: false
-
-      - name: Setup node
-        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
-        with:
-          node-version: 22
-          cache: "npm"
-          cache-dependency-path: ./web/package-lock.json
-
-      - name: Install dependencies
-        working-directory: web
-        run: npm ci
-
-      - name: Build Storybook
-        working-directory: web
-        run: npm run storybook:build
-
-      - name: Deploy to Vercel (Production)
-        working-directory: web
-        run: npx --yes "$VERCEL_CLI" deploy storybook-static/ --prod --yes --token="$VERCEL_TOKEN"
-
-  notify-slack-on-failure:
-    needs: Deploy-Storybook
-    if: always() && needs.Deploy-Storybook.result == 'failure'
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v4
-        with:
-          persist-credentials: false
-          sparse-checkout: .github/actions/slack-notify
-
-      - name: Send Slack notification
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.MONITOR_DEPLOYMENTS_WEBHOOK }}
-          failed-jobs: "• Deploy-Storybook"
-          title: "🚨 Storybook Deploy Failed"

.github/workflows/zizmor.yml

@@ -24,7 +24,7 @@ jobs:
           persist-credentials: false
 
       - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"

.pre-commit-config.yaml

@@ -119,11 +119,10 @@ repos:
           ]
 
   - repo: https://github.com/golangci/golangci-lint
-    rev: 5d1e709b7be35cb2025444e19de266b056b7b7ee # frozen: v2.10.1
+    rev: 9f61b0f53f80672872fced07b6874397c3ed197b # frozen: v2.7.2
     hooks:
       - id: golangci-lint
-        language_version: "1.26.0"
-        entry: bash -c "find . -name go.mod -not -path './.venv/*' -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"
+        entry: bash -c "find tools/ -name go.mod -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.

.vscode/env_template.txt

@@ -7,9 +7,6 @@
 
 
 AUTH_TYPE=basic
-# Recommended for basic auth - used for signing password reset and verification tokens
-# Generate a secure value with: openssl rand -hex 32
-USER_AUTH_SECRET=""
 DEV_MODE=true
 
 

.vscode/launch.json

@@ -40,7 +40,19 @@
       }
     },
     {
-      "name": "Celery",
+      "name": "Celery (lightweight mode)",
+      "configurations": [
+        "Celery primary",
+        "Celery background",
+        "Celery beat"
+      ],
+      "presentation": {
+        "group": "1"
+      },
+      "stopAll": true
+    },
+    {
+      "name": "Celery (standard mode)",
       "configurations": [
         "Celery primary",
         "Celery light",
@@ -241,6 +253,35 @@
       },
       "consoleTitle": "Celery light Console"
     },
+    {
+      "name": "Celery background",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "celery",
+      "cwd": "${workspaceFolder}/backend",
+      "envFile": "${workspaceFolder}/.vscode/.env",
+      "env": {
+        "LOG_LEVEL": "INFO",
+        "PYTHONUNBUFFERED": "1",
+        "PYTHONPATH": "."
+      },
+      "args": [
+        "-A",
+        "onyx.background.celery.versioned_apps.background",
+        "worker",
+        "--pool=threads",
+        "--concurrency=20",
+        "--prefetch-multiplier=4",
+        "--loglevel=INFO",
+        "--hostname=background@%n",
+        "-Q",
+        "vespa_metadata_sync,connector_deletion,doc_permissions_upsert,checkpoint_cleanup,index_attempt_cleanup,docprocessing,connector_doc_fetching,connector_pruning,connector_doc_permissions_sync,connector_external_group_sync,csv_generation,kg_processing,monitoring,user_file_processing,user_file_project_sync,user_file_delete,opensearch_migration"
+      ],
+      "presentation": {
+        "group": "2"
+      },
+      "consoleTitle": "Celery background Console"
+    },
     {
       "name": "Celery heavy",
       "type": "debugpy",
@@ -485,6 +526,21 @@
         "group": "3"
       }
     },
+    {
+      "name": "Clear and Restart OpenSearch Container",
+      // Generic debugger type, required arg but has no bearing on bash.
+      "type": "node",
+      "request": "launch",
+      "runtimeExecutable": "bash",
+      "runtimeArgs": [
+        "${workspaceFolder}/backend/scripts/restart_opensearch_container.sh"
+      ],
+      "cwd": "${workspaceFolder}",
+      "console": "integratedTerminal",
+      "presentation": {
+        "group": "3"
+      }
+    },
     {
       "name": "Eval CLI",
       "type": "debugpy",

AGENTS.md

@@ -86,6 +86,37 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work
      - Monitoring tasks (every 5 minutes)
      - Cleanup tasks (hourly)
 
+#### Worker Deployment Modes
+
+Onyx supports two deployment modes for background workers, controlled by the `USE_LIGHTWEIGHT_BACKGROUND_WORKER` environment variable:
+
+**Lightweight Mode** (default, `USE_LIGHTWEIGHT_BACKGROUND_WORKER=true`):
+
+- Runs a single consolidated `background` worker that handles all background tasks:
+  - Light worker tasks (Vespa operations, permissions sync, deletion)
+  - Document processing (indexing pipeline)
+  - Document fetching (connector data retrieval)
+  - Pruning operations (from `heavy` worker)
+  - Knowledge graph processing (from `kg_processing` worker)
+  - Monitoring tasks (from `monitoring` worker)
+  - User file processing (from `user_file_processing` worker)
+- Lower resource footprint (fewer worker processes)
+- Suitable for smaller deployments or development environments
+- Default concurrency: 20 threads (increased to handle combined workload)
+
+**Standard Mode** (`USE_LIGHTWEIGHT_BACKGROUND_WORKER=false`):
+
+- Runs separate specialized workers as documented above (light, docprocessing, docfetching, heavy, kg_processing, monitoring, user_file_processing)
+- Better isolation and scalability
+- Can scale individual workers independently based on workload
+- Suitable for production deployments with higher load
+
+The deployment mode affects:
+
+- **Backend**: Worker processes spawned by supervisord or dev scripts
+- **Helm**: Which Kubernetes deployments are created
+- **Dev Environment**: Which workers `dev_run_background_jobs.py` spawns
+
 #### Key Features
 
 - **Thread-based Workers**: All workers use thread pools (not processes) for stability
@@ -104,10 +135,6 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work
 
 - Always use `@shared_task` rather than `@celery_app`
 - Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks`
-- Never enqueue a task without an expiration. Always supply `expires=` when
-  sending tasks, either from the beat schedule or directly from another task. It
-  should never be acceptable to submit code which enqueues tasks without an
-  expiration, as doing so can lead to unbounded task queue growth.
 
 **Defining APIs**:
 When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the
@@ -544,8 +571,6 @@ To run them:
 npx playwright test <TEST_NAME>
 ```
 
-For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`.
-
 ## Logs
 
 When (1) writing integration tests or (2) doing live tests (e.g. curl / playwright) you can get access
@@ -592,45 +617,6 @@ Keep it high level. You can reference certain files or functions though.
 
 Before writing your plan, make sure to do research. Explore the relevant sections in the codebase.
 
-## Error Handling
-
-**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`.
-Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.**
-
-A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard
-`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error
-handling consistent across the entire backend.
-
-```python
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
-
-# ✅ Good
-raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found")
-
-# ✅ Good — no extra message needed
-raise OnyxError(OnyxErrorCode.UNAUTHENTICATED)
-
-# ✅ Good — upstream service with dynamic status code
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status)
-
-# ❌ Bad — using HTTPException directly
-raise HTTPException(status_code=404, detail="Session not found")
-
-# ❌ Bad — starlette constant
-raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
-```
-
-Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error
-category is needed, add it there first — do not invent ad-hoc codes.
-
-**Upstream service errors:** When forwarding errors from an upstream service where the HTTP
-status code is dynamic (comes from the upstream response), use `status_code_override`:
-
-```python
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code)
-```
-
 ## Best Practices
 
 In addition to the other content in this file, best practices for contributing

backend/Dockerfile

@@ -46,9 +46,7 @@ RUN apt-get update && \
         pkg-config \
         gcc \
         nano \
-        vim \
-        libjemalloc2 \
-        && \
+        vim && \
     rm -rf /var/lib/apt/lists/* && \
     apt-get clean
 
@@ -143,7 +141,6 @@ COPY --chown=onyx:onyx ./scripts/debugging /app/scripts/debugging
 COPY --chown=onyx:onyx ./scripts/force_delete_connector_by_id.py /app/scripts/force_delete_connector_by_id.py
 COPY --chown=onyx:onyx ./scripts/supervisord_entrypoint.sh /app/scripts/supervisord_entrypoint.sh
 COPY --chown=onyx:onyx ./scripts/setup_craft_templates.sh /app/scripts/setup_craft_templates.sh
-COPY --chown=onyx:onyx ./scripts/reencrypt_secrets.py /app/scripts/reencrypt_secrets.py
 RUN chmod +x /app/scripts/supervisord_entrypoint.sh /app/scripts/setup_craft_templates.sh
 
 # Run Craft template setup at build time when ENABLE_CRAFT=true
@@ -167,13 +164,6 @@ ENV PYTHONPATH=/app
 ARG ONYX_VERSION=0.0.0-dev
 ENV ONYX_VERSION=${ONYX_VERSION}
 
-# Use jemalloc instead of glibc malloc to reduce memory fragmentation
-# in long-running Python processes (API server, Celery workers).
-# The soname is architecture-independent; the dynamic linker resolves
-# the correct path from standard library directories.
-# Placed after all RUN steps so build-time processes are unaffected.
-ENV LD_PRELOAD=libjemalloc.so.2
-
 # Default command which does nothing
 # This container is used by api server and background which specify their own CMD
 CMD ["tail", "-f", "/dev/null"]

backend/alembic/versions/2664261bfaab_add_cache_store_table.py

@@ -1,37 +0,0 @@
-"""add cache_store table
-
-Revision ID: 2664261bfaab
-Revises: 4a1e4b1c89d2
-Create Date: 2026-02-27 00:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-# revision identifiers, used by Alembic.
-revision = "2664261bfaab"
-down_revision = "4a1e4b1c89d2"
-branch_labels: None = None
-depends_on: None = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "cache_store",
-        sa.Column("key", sa.String(), nullable=False),
-        sa.Column("value", sa.LargeBinary(), nullable=True),
-        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
-        sa.PrimaryKeyConstraint("key"),
-    )
-    op.create_index(
-        "ix_cache_store_expires",
-        "cache_store",
-        ["expires_at"],
-        postgresql_where=sa.text("expires_at IS NOT NULL"),
-    )
-
-
-def downgrade() -> None:
-    op.drop_index("ix_cache_store_expires", table_name="cache_store")
-    op.drop_table("cache_store")

backend/alembic/versions/27fb147a843f_add_timestamps_to_user_table.py

@@ -1,43 +0,0 @@
-"""add timestamps to user table
-
-Revision ID: 27fb147a843f
-Revises: b5c4d7e8f9a1
-Create Date: 2026-03-08 17:18:40.828644
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "27fb147a843f"
-down_revision = "b5c4d7e8f9a1"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "user",
-        sa.Column(
-            "created_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "updated_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("user", "updated_at")
-    op.drop_column("user", "created_at")

backend/alembic/versions/689433b0d8de_add_hook_and_hook_execution_log_tables.py

@@ -1,104 +0,0 @@
-"""add_hook_and_hook_execution_log_tables
-
-Revision ID: 689433b0d8de
-Revises: 93a2e195e25c
-Create Date: 2026-03-13 11:25:06.547474
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.dialects.postgresql import UUID as PGUUID
-
-
-# revision identifiers, used by Alembic.
-revision = "689433b0d8de"
-down_revision = "93a2e195e25c"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "hook",
-        sa.Column("id", sa.Integer(), nullable=False),
-        sa.Column("name", sa.String(), nullable=False),
-        sa.Column(
-            "hook_point",
-            sa.Enum("document_ingestion", "query_processing", native_enum=False),
-            nullable=False,
-        ),
-        sa.Column("endpoint_url", sa.Text(), nullable=True),
-        sa.Column("api_key", sa.LargeBinary(), nullable=True),
-        sa.Column("is_reachable", sa.Boolean(), nullable=True),
-        sa.Column(
-            "fail_strategy",
-            sa.Enum("hard", "soft", native_enum=False),
-            nullable=False,
-            server_default="hard",
-        ),
-        sa.Column("timeout_seconds", sa.Float(), nullable=False, server_default="30.0"),
-        sa.Column(
-            "is_active", sa.Boolean(), nullable=False, server_default=sa.text("false")
-        ),
-        sa.Column(
-            "deleted", sa.Boolean(), nullable=False, server_default=sa.text("false")
-        ),
-        sa.Column("creator_id", PGUUID(as_uuid=True), nullable=True),
-        sa.Column(
-            "created_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.text("now()"),
-            nullable=False,
-        ),
-        sa.Column(
-            "updated_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.text("now()"),
-            nullable=False,
-        ),
-        sa.ForeignKeyConstraint(["creator_id"], ["user.id"], ondelete="SET NULL"),
-        sa.PrimaryKeyConstraint("id"),
-    )
-    op.create_index(
-        "ix_hook_one_active_per_point",
-        "hook",
-        ["hook_point"],
-        unique=True,
-        postgresql_where=sa.text("is_active = true AND deleted = false"),
-    )
-
-    op.create_table(
-        "hook_execution_log",
-        sa.Column("id", sa.Integer(), nullable=False),
-        sa.Column("hook_id", sa.Integer(), nullable=False),
-        sa.Column(
-            "hook_point",
-            sa.Enum("document_ingestion", "query_processing", native_enum=False),
-            nullable=False,
-        ),
-        sa.Column("error_message", sa.Text(), nullable=True),
-        sa.Column("status_code", sa.Integer(), nullable=True),
-        sa.Column("duration_ms", sa.Integer(), nullable=True),
-        sa.Column(
-            "created_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.text("now()"),
-            nullable=False,
-        ),
-        sa.ForeignKeyConstraint(["hook_id"], ["hook.id"], ondelete="CASCADE"),
-        sa.PrimaryKeyConstraint("id"),
-    )
-    op.create_index("ix_hook_execution_log_hook_id", "hook_execution_log", ["hook_id"])
-    op.create_index(
-        "ix_hook_execution_log_created_at", "hook_execution_log", ["created_at"]
-    )
-
-
-def downgrade() -> None:
-    op.drop_index("ix_hook_execution_log_created_at", table_name="hook_execution_log")
-    op.drop_index("ix_hook_execution_log_hook_id", table_name="hook_execution_log")
-    op.drop_table("hook_execution_log")
-
-    op.drop_index("ix_hook_one_active_per_point", table_name="hook")
-    op.drop_table("hook")

backend/alembic/versions/93a2e195e25c_add_voice_provider_and_user_voice_prefs.py

@@ -1,117 +0,0 @@
-"""add_voice_provider_and_user_voice_prefs
-
-Revision ID: 93a2e195e25c
-Revises: 27fb147a843f
-Create Date: 2026-02-23 15:16:39.507304
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy import column
-from sqlalchemy import true
-from sqlalchemy.dialects import postgresql
-
-
-# revision identifiers, used by Alembic.
-revision = "93a2e195e25c"
-down_revision = "27fb147a843f"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Create voice_provider table
-    op.create_table(
-        "voice_provider",
-        sa.Column("id", sa.Integer(), primary_key=True),
-        sa.Column("name", sa.String(), unique=True, nullable=False),
-        sa.Column("provider_type", sa.String(), nullable=False),
-        sa.Column("api_key", sa.LargeBinary(), nullable=True),
-        sa.Column("api_base", sa.String(), nullable=True),
-        sa.Column("custom_config", postgresql.JSONB(), nullable=True),
-        sa.Column("stt_model", sa.String(), nullable=True),
-        sa.Column("tts_model", sa.String(), nullable=True),
-        sa.Column("default_voice", sa.String(), nullable=True),
-        sa.Column(
-            "is_default_stt", sa.Boolean(), nullable=False, server_default="false"
-        ),
-        sa.Column(
-            "is_default_tts", sa.Boolean(), nullable=False, server_default="false"
-        ),
-        sa.Column("deleted", sa.Boolean(), nullable=False, server_default="false"),
-        sa.Column(
-            "time_created",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-        sa.Column(
-            "time_updated",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            onupdate=sa.func.now(),
-            nullable=False,
-        ),
-    )
-
-    # Add partial unique indexes to enforce only one default STT/TTS provider
-    op.create_index(
-        "ix_voice_provider_one_default_stt",
-        "voice_provider",
-        ["is_default_stt"],
-        unique=True,
-        postgresql_where=column("is_default_stt") == true(),
-    )
-    op.create_index(
-        "ix_voice_provider_one_default_tts",
-        "voice_provider",
-        ["is_default_tts"],
-        unique=True,
-        postgresql_where=column("is_default_tts") == true(),
-    )
-
-    # Add voice preference columns to user table
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_auto_send",
-            sa.Boolean(),
-            default=False,
-            nullable=False,
-            server_default="false",
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_auto_playback",
-            sa.Boolean(),
-            default=False,
-            nullable=False,
-            server_default="false",
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_playback_speed",
-            sa.Float(),
-            default=1.0,
-            nullable=False,
-            server_default="1.0",
-        ),
-    )
-
-
-def downgrade() -> None:
-    # Remove user voice preference columns
-    op.drop_column("user", "voice_playback_speed")
-    op.drop_column("user", "voice_auto_playback")
-    op.drop_column("user", "voice_auto_send")
-
-    op.drop_index("ix_voice_provider_one_default_tts", table_name="voice_provider")
-    op.drop_index("ix_voice_provider_one_default_stt", table_name="voice_provider")
-
-    # Drop voice_provider table
-    op.drop_table("voice_provider")

backend/alembic/versions/a3b8d9e2f1c4_make_scim_external_id_nullable.py

@@ -1,34 +0,0 @@
-"""make scim_user_mapping.external_id nullable
-
-Revision ID: a3b8d9e2f1c4
-Revises: 2664261bfaab
-Create Date: 2026-03-02
-
-"""
-
-from alembic import op
-
-
-# revision identifiers, used by Alembic.
-revision = "a3b8d9e2f1c4"
-down_revision = "2664261bfaab"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=True,
-    )
-
-
-def downgrade() -> None:
-    # Delete any rows where external_id is NULL before re-applying NOT NULL
-    op.execute("DELETE FROM scim_user_mapping WHERE external_id IS NULL")
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=False,
-    )

backend/alembic/versions/b5c4d7e8f9a1_add_hierarchy_node_cc_pair_table.py

@@ -1,51 +0,0 @@
-"""add hierarchy_node_by_connector_credential_pair table
-
-Revision ID: b5c4d7e8f9a1
-Revises: a3b8d9e2f1c4
-Create Date: 2026-03-04
-
-"""
-
-import sqlalchemy as sa
-from alembic import op
-
-revision = "b5c4d7e8f9a1"
-down_revision = "a3b8d9e2f1c4"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "hierarchy_node_by_connector_credential_pair",
-        sa.Column("hierarchy_node_id", sa.Integer(), nullable=False),
-        sa.Column("connector_id", sa.Integer(), nullable=False),
-        sa.Column("credential_id", sa.Integer(), nullable=False),
-        sa.ForeignKeyConstraint(
-            ["hierarchy_node_id"],
-            ["hierarchy_node.id"],
-            ondelete="CASCADE",
-        ),
-        sa.ForeignKeyConstraint(
-            ["connector_id", "credential_id"],
-            [
-                "connector_credential_pair.connector_id",
-                "connector_credential_pair.credential_id",
-            ],
-            ondelete="CASCADE",
-        ),
-        sa.PrimaryKeyConstraint("hierarchy_node_id", "connector_id", "credential_id"),
-    )
-    op.create_index(
-        "ix_hierarchy_node_cc_pair_connector_credential",
-        "hierarchy_node_by_connector_credential_pair",
-        ["connector_id", "credential_id"],
-    )
-
-
-def downgrade() -> None:
-    op.drop_index(
-        "ix_hierarchy_node_cc_pair_connector_credential",
-        table_name="hierarchy_node_by_connector_credential_pair",
-    )
-    op.drop_table("hierarchy_node_by_connector_credential_pair")

backend/alembic_tenants/versions/3b9f09038764_add_read_only_kg_user.py

@@ -11,6 +11,7 @@ from sqlalchemy import text
 from alembic import op
 from onyx.configs.app_configs import DB_READONLY_PASSWORD
 from onyx.configs.app_configs import DB_READONLY_USER
+from shared_configs.configs import MULTI_TENANT
 
 
 # revision identifiers, used by Alembic.
@@ -21,52 +22,59 @@ depends_on = None
 
 
 def upgrade() -> None:
-    # Enable pg_trgm extension if not already enabled
-    op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
+    if MULTI_TENANT:
 
-    # Create the read-only db user if it does not already exist.
-    if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
-        raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+        # Enable pg_trgm extension if not already enabled
+        op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
 
-    op.execute(
-        text(
-            f"""
-            DO $$
-            BEGIN
-                -- Check if the read-only user already exists
-                IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                    -- Create the read-only user with the specified password
-                    EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
-                    -- First revoke all privileges to ensure a clean slate
-                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                    -- Grant only the CONNECT privilege to allow the user to connect to the database
-                    -- but not perform any operations without additional specific grants
-                    EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
-                END IF;
-            END
-            $$;
-            """
+        # Create read-only db user here only in multi-tenant mode. For single-tenant mode,
+        # the user is created in the standard migration.
+        if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
+            raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+
+        op.execute(
+            text(
+                f"""
+                DO $$
+                BEGIN
+                    -- Check if the read-only user already exists
+                    IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                        -- Create the read-only user with the specified password
+                        EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
+                        -- First revoke all privileges to ensure a clean slate
+                        EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                        -- Grant only the CONNECT privilege to allow the user to connect to the database
+                        -- but not perform any operations without additional specific grants
+                        EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
+                    END IF;
+                END
+                $$;
+                """
+            )
         )
-    )
 
 
 def downgrade() -> None:
-    op.execute(
-        text(
-            f"""
-        DO $$
-        BEGIN
-            IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                -- First revoke all privileges from the database
-                EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                -- Then revoke all privileges from the public schema
-                EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
-                -- Then drop the user
-                EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
-            END IF;
-        END
-        $$;
-    """
+    if MULTI_TENANT:
+        # Drop read-only db user here only in single tenant mode. For multi-tenant mode,
+        # the user is dropped in the alembic_tenants migration.
+
+        op.execute(
+            text(
+                f"""
+            DO $$
+            BEGIN
+                IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                    -- First revoke all privileges from the database
+                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                    -- Then revoke all privileges from the public schema
+                    EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
+                    -- Then drop the user
+                    EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
+                END IF;
+            END
+            $$;
+        """
+            )
         )
-    )
-    op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))
+        op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))

backend/ee/onyx/access/access.py

@@ -9,15 +9,12 @@ from onyx.access.access import (
     _get_access_for_documents as get_access_for_documents_without_groups,
 )
 from onyx.access.access import _get_acl_for_user as get_acl_for_user_without_groups
-from onyx.access.access import collect_user_file_access
 from onyx.access.models import DocumentAccess
 from onyx.access.utils import prefix_external_group
 from onyx.access.utils import prefix_user_group
 from onyx.db.document import get_document_sources
 from onyx.db.document import get_documents_by_ids
 from onyx.db.models import User
-from onyx.db.models import UserFile
-from onyx.db.user_file import fetch_user_files_with_access_relationships
 from onyx.utils.logger import setup_logger
 
 
@@ -119,68 +116,6 @@ def _get_access_for_documents(
     return access_map
 
 
-def _collect_user_file_group_names(user_file: UserFile) -> set[str]:
-    """Extract user-group names from the already-loaded Persona.groups
-    relationships on a UserFile (skipping deleted personas)."""
-    groups: set[str] = set()
-    for persona in user_file.assistants:
-        if persona.deleted:
-            continue
-        for group in persona.groups:
-            groups.add(group.name)
-    return groups
-
-
-def get_access_for_user_files_impl(
-    user_file_ids: list[str],
-    db_session: Session,
-) -> dict[str, DocumentAccess]:
-    """EE version: extends the MIT user file ACL with user group names
-    from personas shared via user groups.
-
-    Uses a single DB query (via fetch_user_files_with_access_relationships)
-    that eagerly loads both the MIT-needed and EE-needed relationships.
-
-    NOTE: is imported in onyx.access.access by `fetch_versioned_implementation`
-    DO NOT REMOVE."""
-    user_files = fetch_user_files_with_access_relationships(
-        user_file_ids, db_session, eager_load_groups=True
-    )
-    return build_access_for_user_files_impl(user_files)
-
-
-def build_access_for_user_files_impl(
-    user_files: list[UserFile],
-) -> dict[str, DocumentAccess]:
-    """EE version: works on pre-loaded UserFile objects.
-    Expects Persona.groups to be eagerly loaded.
-
-    NOTE: is imported in onyx.access.access by `fetch_versioned_implementation`
-    DO NOT REMOVE."""
-    result: dict[str, DocumentAccess] = {}
-    for user_file in user_files:
-        if user_file.user is None:
-            result[str(user_file.id)] = DocumentAccess.build(
-                user_emails=[],
-                user_groups=[],
-                is_public=True,
-                external_user_emails=[],
-                external_user_group_ids=[],
-            )
-            continue
-
-        emails, is_public = collect_user_file_access(user_file)
-        group_names = _collect_user_file_group_names(user_file)
-        result[str(user_file.id)] = DocumentAccess.build(
-            user_emails=list(emails),
-            user_groups=list(group_names),
-            is_public=is_public,
-            external_user_emails=[],
-            external_user_group_ids=[],
-        )
-    return result
-
-
 def _get_acl_for_user(user: User, db_session: Session) -> set[str]:
     """Returns a list of ACL entries that the user has access to. This is meant to be
     used downstream to filter out documents that the user does not have access to. The

backend/ee/onyx/auth/users.py

@@ -1,4 +1,3 @@
-import os
 from datetime import datetime
 
 import jwt
@@ -21,13 +20,7 @@ logger = setup_logger()
 
 
 def verify_auth_setting() -> None:
-    # All the Auth flows are valid for EE version, but warn about deprecated 'disabled'
-    raw_auth_type = (os.environ.get("AUTH_TYPE") or "").lower()
-    if raw_auth_type == "disabled":
-        logger.warning(
-            "AUTH_TYPE='disabled' is no longer supported. "
-            "Using 'basic' instead. Please update your configuration."
-        )
+    # All the Auth flows are valid for EE version
     logger.notice(f"Using Auth Type: {AUTH_TYPE.value}")
 
 

backend/ee/onyx/background/celery/apps/background.py

@@ -0,0 +1,15 @@
+from onyx.background.celery.apps import app_base
+from onyx.background.celery.apps.background import celery_app
+
+
+celery_app.autodiscover_tasks(
+    app_base.filter_task_modules(
+        [
+            "ee.onyx.background.celery.tasks.doc_permission_syncing",
+            "ee.onyx.background.celery.tasks.external_group_syncing",
+            "ee.onyx.background.celery.tasks.cleanup",
+            "ee.onyx.background.celery.tasks.tenant_provisioning",
+            "ee.onyx.background.celery.tasks.query_history",
+        ]
+    )
+)

backend/ee/onyx/db/hierarchy.py

@@ -18,7 +18,7 @@ from onyx.db.models import HierarchyNode
 
 
 def _build_hierarchy_access_filter(
-    user_email: str,
+    user_email: str | None,
     external_group_ids: list[str],
 ) -> ColumnElement[bool]:
     """Build SQLAlchemy filter for hierarchy node access.
@@ -43,7 +43,7 @@ def _build_hierarchy_access_filter(
 def _get_accessible_hierarchy_nodes_for_source(
     db_session: Session,
     source: DocumentSource,
-    user_email: str,
+    user_email: str | None,
     external_group_ids: list[str],
 ) -> list[HierarchyNode]:
     """

backend/ee/onyx/db/license.py

@@ -11,10 +11,11 @@ from ee.onyx.server.license.models import LicenseMetadata
 from ee.onyx.server.license.models import LicensePayload
 from ee.onyx.server.license.models import LicenseSource
 from onyx.auth.schemas import UserRole
-from onyx.cache.factory import get_cache_backend
 from onyx.configs.constants import ANONYMOUS_USER_EMAIL
 from onyx.db.models import License
 from onyx.db.models import User
+from onyx.redis.redis_pool import get_redis_client
+from onyx.redis.redis_pool import get_redis_replica_client
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.contextvars import get_current_tenant_id
@@ -141,7 +142,7 @@ def get_used_seats(tenant_id: str | None = None) -> int:
 
 def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata | None:
     """
-    Get license metadata from cache.
+    Get license metadata from Redis cache.
 
     Args:
         tenant_id: Tenant ID (for multi-tenant deployments)
@@ -149,34 +150,38 @@ def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata
     Returns:
         LicenseMetadata if cached, None otherwise
     """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cached = cache.get(LICENSE_METADATA_KEY)
-    if not cached:
-        return None
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_replica_client(tenant_id=tenant)
 
-    try:
-        cached_str = (
-            cached.decode("utf-8") if isinstance(cached, bytes) else str(cached)
-        )
-        return LicenseMetadata.model_validate_json(cached_str)
-    except Exception as e:
-        logger.warning(f"Failed to parse cached license metadata: {e}")
-        return None
+    cached = redis_client.get(LICENSE_METADATA_KEY)
+    if cached:
+        try:
+            cached_str: str
+            if isinstance(cached, bytes):
+                cached_str = cached.decode("utf-8")
+            else:
+                cached_str = str(cached)
+            return LicenseMetadata.model_validate_json(cached_str)
+        except Exception as e:
+            logger.warning(f"Failed to parse cached license metadata: {e}")
+            return None
+    return None
 
 
 def invalidate_license_cache(tenant_id: str | None = None) -> None:
     """
     Invalidate the license metadata cache (not the license itself).
 
-    Deletes the cached LicenseMetadata. The actual license in the database
-    is not affected. Delete is idempotent — if the key doesn't exist, this
-    is a no-op.
+    This deletes the cached LicenseMetadata from Redis. The actual license
+    in the database is not affected. Redis delete is idempotent - if the
+    key doesn't exist, this is a no-op.
 
     Args:
         tenant_id: Tenant ID (for multi-tenant deployments)
     """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cache.delete(LICENSE_METADATA_KEY)
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_client(tenant_id=tenant)
+    redis_client.delete(LICENSE_METADATA_KEY)
     logger.info("License cache invalidated")
 
 
@@ -187,7 +192,7 @@ def update_license_cache(
     tenant_id: str | None = None,
 ) -> LicenseMetadata:
     """
-    Update the cache with license metadata.
+    Update the Redis cache with license metadata.
 
     We cache all license statuses (ACTIVE, GRACE_PERIOD, GATED_ACCESS) because:
     1. Frontend needs status to show appropriate UI/banners
@@ -206,7 +211,7 @@ def update_license_cache(
     from ee.onyx.utils.license import get_license_status
 
     tenant = tenant_id or get_current_tenant_id()
-    cache = get_cache_backend(tenant_id=tenant_id)
+    redis_client = get_redis_client(tenant_id=tenant)
 
     used_seats = get_used_seats(tenant)
     status = get_license_status(payload, grace_period_end)
@@ -225,7 +230,7 @@ def update_license_cache(
         stripe_subscription_id=payload.stripe_subscription_id,
     )
 
-    cache.set(
+    redis_client.set(
         LICENSE_METADATA_KEY,
         metadata.model_dump_json(),
         ex=LICENSE_CACHE_TTL_SECONDS,

backend/ee/onyx/db/persona.py

@@ -7,7 +7,6 @@ from onyx.db.models import Persona
 from onyx.db.models import Persona__User
 from onyx.db.models import Persona__UserGroup
 from onyx.db.notification import create_notification
-from onyx.db.persona import mark_persona_user_files_for_sync
 from onyx.server.features.persona.models import PersonaSharedNotificationData
 
 
@@ -27,9 +26,7 @@ def update_persona_access(
 
     NOTE: Callers are responsible for committing."""
 
-    needs_sync = False
     if is_public is not None:
-        needs_sync = True
         persona = db_session.query(Persona).filter(Persona.id == persona_id).first()
         if persona:
             persona.is_public = is_public
@@ -38,7 +35,6 @@ def update_persona_access(
     # and a non-empty list means "replace with these shares".
 
     if user_ids is not None:
-        needs_sync = True
         db_session.query(Persona__User).filter(
             Persona__User.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -58,7 +54,6 @@ def update_persona_access(
                 )
 
     if group_ids is not None:
-        needs_sync = True
         db_session.query(Persona__UserGroup).filter(
             Persona__UserGroup.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -68,7 +63,3 @@ def update_persona_access(
             db_session.add(
                 Persona__UserGroup(persona_id=persona_id, user_group_id=group_id)
             )
-
-    # When sharing changes, user file ACLs need to be updated in the vector DB
-    if needs_sync:
-        mark_persona_user_files_for_sync(persona_id, db_session)

backend/ee/onyx/db/scim.py

@@ -126,16 +126,12 @@ class ScimDAL(DAL):
 
     def create_user_mapping(
         self,
-        external_id: str | None,
+        external_id: str,
         user_id: UUID,
         scim_username: str | None = None,
         fields: ScimMappingFields | None = None,
     ) -> ScimUserMapping:
-        """Create a SCIM mapping for a user.
-
-        ``external_id`` may be ``None`` when the IdP omits it (RFC 7643
-        allows this). The mapping still marks the user as SCIM-managed.
-        """
+        """Create a mapping between a SCIM externalId and an Onyx user."""
         f = fields or ScimMappingFields()
         mapping = ScimUserMapping(
             external_id=external_id,
@@ -274,13 +270,8 @@ class ScimDAL(DAL):
         Raises:
             ValueError: If the filter uses an unsupported attribute.
         """
-        # Inner-join with ScimUserMapping so only SCIM-managed users appear.
-        # Pre-existing system accounts (anonymous, admin, etc.) are excluded
-        # unless they were explicitly linked via SCIM provisioning.
-        query = (
-            select(User)
-            .join(ScimUserMapping, ScimUserMapping.user_id == User.id)
-            .where(User.role.notin_([UserRole.SLACK_USER, UserRole.EXT_PERM_USER]))
+        query = select(User).where(
+            User.role.notin_([UserRole.SLACK_USER, UserRole.EXT_PERM_USER])
         )
 
         if scim_filter:
@@ -330,37 +321,34 @@ class ScimDAL(DAL):
         scim_username: str | None = None,
         fields: ScimMappingFields | None = None,
     ) -> None:
-        """Sync the SCIM mapping for a user.
-
-        If a mapping already exists, its fields are updated (including
-        setting ``external_id`` to ``None`` when the IdP omits it).
-        If no mapping exists and ``new_external_id`` is provided, a new
-        mapping is created.  A mapping is never deleted here — SCIM-managed
-        users must retain their mapping to remain visible in ``GET /Users``.
+        """Create, update, or delete the external ID mapping for a user.
 
         When *fields* is provided, all mapping fields are written
         unconditionally — including ``None`` values — so that a caller can
         clear a previously-set field (e.g. removing a department).
         """
         mapping = self.get_user_mapping_by_user_id(user_id)
-        if mapping:
-            if mapping.external_id != new_external_id:
-                mapping.external_id = new_external_id
-            if scim_username is not None:
-                mapping.scim_username = scim_username
-            if fields is not None:
-                mapping.department = fields.department
-                mapping.manager = fields.manager
-                mapping.given_name = fields.given_name
-                mapping.family_name = fields.family_name
-                mapping.scim_emails_json = fields.scim_emails_json
-        elif new_external_id:
-            self.create_user_mapping(
-                external_id=new_external_id,
-                user_id=user_id,
-                scim_username=scim_username,
-                fields=fields,
-            )
+        if new_external_id:
+            if mapping:
+                if mapping.external_id != new_external_id:
+                    mapping.external_id = new_external_id
+                if scim_username is not None:
+                    mapping.scim_username = scim_username
+                if fields is not None:
+                    mapping.department = fields.department
+                    mapping.manager = fields.manager
+                    mapping.given_name = fields.given_name
+                    mapping.family_name = fields.family_name
+                    mapping.scim_emails_json = fields.scim_emails_json
+            else:
+                self.create_user_mapping(
+                    external_id=new_external_id,
+                    user_id=user_id,
+                    scim_username=scim_username,
+                    fields=fields,
+                )
+        elif mapping:
+            self.delete_user_mapping(mapping.id)
 
     def _get_user_mappings_batch(
         self, user_ids: list[UUID]

backend/ee/onyx/db/user_group.py

@@ -15,7 +15,6 @@ from sqlalchemy.orm import Session
 from ee.onyx.server.user_group.models import SetCuratorRequest
 from ee.onyx.server.user_group.models import UserGroupCreate
 from ee.onyx.server.user_group.models import UserGroupUpdate
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.db.connector_credential_pair import get_connector_credential_pair_from_id
 from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
@@ -472,9 +471,7 @@ def _add_user_group__cc_pair_relationships__no_commit(
 
 def insert_user_group(db_session: Session, user_group: UserGroupCreate) -> UserGroup:
     db_user_group = UserGroup(
-        name=user_group.name,
-        time_last_modified_by_user=func.now(),
-        is_up_to_date=DISABLE_VECTOR_DB,
+        name=user_group.name, time_last_modified_by_user=func.now()
     )
     db_session.add(db_user_group)
     db_session.flush()  # give the group an ID
@@ -777,7 +774,8 @@ def update_user_group(
             cc_pair_ids=user_group_update.cc_pair_ids,
         )
 
-    if cc_pairs_updated and not DISABLE_VECTOR_DB:
+    # only needs to sync with Vespa if the cc_pairs have been updated
+    if cc_pairs_updated:
         db_user_group.is_up_to_date = False
 
     removed_users = db_session.scalars(

backend/ee/onyx/external_permissions/google_drive/doc_sync.py

@@ -68,7 +68,6 @@ def get_external_access_for_raw_gdrive_file(
     company_domain: str,
     retriever_drive_service: GoogleDriveService | None,
     admin_drive_service: GoogleDriveService,
-    fallback_user_email: str,
     add_prefix: bool = False,
 ) -> ExternalAccess:
     """
@@ -80,11 +79,6 @@ def get_external_access_for_raw_gdrive_file(
                 set add_prefix to True so group IDs are prefixed with the source type.
                 When invoked from doc_sync (permission sync), use the default (False)
                 since upsert_document_external_perms handles prefixing.
-    fallback_user_email: When we cannot retrieve any permission info for a file
-                (e.g. externally-owned files where the API returns no permissions
-                and permissions.list returns 403), fall back to granting access
-                to this user. This is typically the impersonated org user whose
-                drive contained the file.
     """
     doc_id = file.get("id")
     if not doc_id:
@@ -123,26 +117,6 @@ def get_external_access_for_raw_gdrive_file(
                 [permissions_list, backup_permissions_list]
             )
 
-    # For externally-owned files, the Drive API may return no permissions
-    # and permissions.list may return 403. In this case, fall back to
-    # granting access to the user who found the file in their drive.
-    # Note, even if other users also have access to this file,
-    # they will not be granted access in Onyx.
-    # We check permissions_list (the final result after all fetch attempts)
-    # rather than the raw fields, because permission_ids may be present
-    # but the actual fetch can still return empty due to a 403.
-    if not permissions_list:
-        logger.info(
-            f"No permission info available for file {doc_id} "
-            f"(likely owned by a user outside of your organization). "
-            f"Falling back to granting access to retriever user: {fallback_user_email}"
-        )
-        return ExternalAccess(
-            external_user_emails={fallback_user_email},
-            external_user_group_ids=set(),
-            is_public=False,
-        )
-
     folder_ids_to_inherit_permissions_from: set[str] = set()
     user_emails: set[str] = set()
     group_emails: set[str] = set()

backend/ee/onyx/external_permissions/jira/group_sync.py

@@ -1,8 +1,6 @@
 from collections.abc import Generator
-from typing import Any
 
 from jira import JIRA
-from jira.exceptions import JIRAError
 
 from ee.onyx.db.external_perm import ExternalUserGroup
 from onyx.connectors.jira.utils import build_jira_client
@@ -11,102 +9,107 @@ from onyx.utils.logger import setup_logger
 
 logger = setup_logger()
 
-_ATLASSIAN_ACCOUNT_TYPE = "atlassian"
-_GROUP_MEMBER_PAGE_SIZE = 50
 
-# The GET /group/member endpoint was introduced in Jira 6.0.
-# Jira versions older than 6.0 do not have group management REST APIs at all.
-_MIN_JIRA_VERSION_FOR_GROUP_MEMBER = "6.0"
-
-
-def _fetch_group_member_page(
+def _get_jira_group_members_email(
     jira_client: JIRA,
     group_name: str,
-    start_at: int,
-) -> dict[str, Any]:
-    """Fetch a single page from the non-deprecated GET /group/member endpoint.
+) -> list[str]:
+    """Get all member emails for a Jira group.
 
-    The old GET /group endpoint (used by jira_client.group_members()) is deprecated
-    and decommissioned in Jira Server 10.3+. This uses the replacement endpoint
-    directly via the library's internal _get_json helper, following the same pattern
-    as enhanced_search_ids / bulk_fetch_issues in connector.py.
-
-    There is an open PR to the library to switch to this endpoint since last year:
-    https://github.com/pycontribs/jira/pull/2356
-    so once it is merged and released, we can switch to using the library function.
+    Filters out app accounts (bots, integrations) and only returns real user emails.
     """
+    emails: list[str] = []
+
     try:
-        return jira_client._get_json(
-            "group/member",
-            params={
-                "groupname": group_name,
-                "includeInactiveUsers": "false",
-                "startAt": start_at,
-                "maxResults": _GROUP_MEMBER_PAGE_SIZE,
-            },
-        )
-    except JIRAError as e:
-        if e.status_code == 404:
-            raise RuntimeError(
-                f"GET /group/member returned 404 for group '{group_name}'. "
-                f"This endpoint requires Jira {_MIN_JIRA_VERSION_FOR_GROUP_MEMBER}+. "
-                f"If you are running a self-hosted Jira instance, please upgrade "
-                f"to at least Jira {_MIN_JIRA_VERSION_FOR_GROUP_MEMBER}."
-            ) from e
-        raise
+        # group_members returns an OrderedDict of account_id -> member_info
+        members = jira_client.group_members(group=group_name)
 
+        if not members:
+            logger.warning(f"No members found for group {group_name}")
+            return emails
 
-def _get_group_member_emails(
-    jira_client: JIRA,
-    group_name: str,
-) -> set[str]:
-    """Get all member emails for a single Jira group.
+        for account_id, member_info in members.items():
+            # member_info is a dict with keys like 'fullname', 'email', 'active'
+            email = member_info.get("email")
 
-    Uses the non-deprecated GET /group/member endpoint which returns full user
-    objects including accountType, so we can filter out app/customer accounts
-    without making separate user() calls.
-    """
-    emails: set[str] = set()
-    start_at = 0
-
-    while True:
-        try:
-            page = _fetch_group_member_page(jira_client, group_name, start_at)
-        except Exception as e:
-            logger.error(f"Error fetching members for group {group_name}: {e}")
-            raise
-
-        members: list[dict[str, Any]] = page.get("values", [])
-        for member in members:
-            account_type = member.get("accountType")
-            # On Jira DC < 9.0, accountType is absent; include those users.
-            # On Cloud / DC 9.0+, filter to real user accounts only.
-            if account_type is not None and account_type != _ATLASSIAN_ACCOUNT_TYPE:
-                continue
-
-            email = member.get("emailAddress")
-            if email:
-                emails.add(email)
+            # Skip "hidden" emails - these are typically app accounts
+            if email and email != "hidden":
+                emails.append(email)
             else:
-                logger.warning(
-                    f"Atlassian user {member.get('accountId', 'unknown')} "
-                    f"in group {group_name} has no visible email address"
-                )
+                # For cloud, we might need to fetch user details separately
+                try:
+                    user = jira_client.user(id=account_id)
 
-        if page.get("isLast", True) or not members:
-            break
-        start_at += len(members)
+                    # Skip app accounts (bots, integrations, etc.)
+                    if hasattr(user, "accountType") and user.accountType == "app":
+                        logger.info(
+                            f"Skipping app account {account_id} for group {group_name}"
+                        )
+                        continue
+
+                    if hasattr(user, "emailAddress") and user.emailAddress:
+                        emails.append(user.emailAddress)
+                    else:
+                        logger.warning(f"User {account_id} has no email address")
+                except Exception as e:
+                    logger.warning(
+                        f"Could not fetch email for user {account_id} in group {group_name}: {e}"
+                    )
+
+    except Exception as e:
+        logger.error(f"Error fetching members for group {group_name}: {e}")
 
     return emails
 
 
+def _build_group_member_email_map(
+    jira_client: JIRA,
+) -> dict[str, set[str]]:
+    """Build a map of group names to member emails."""
+    group_member_emails: dict[str, set[str]] = {}
+
+    try:
+        # Get all groups from Jira - returns a list of group name strings
+        group_names = jira_client.groups()
+
+        if not group_names:
+            logger.warning("No groups found in Jira")
+            return group_member_emails
+
+        logger.info(f"Found {len(group_names)} groups in Jira")
+
+        for group_name in group_names:
+            if not group_name:
+                continue
+
+            member_emails = _get_jira_group_members_email(
+                jira_client=jira_client,
+                group_name=group_name,
+            )
+
+            if member_emails:
+                group_member_emails[group_name] = set(member_emails)
+                logger.debug(
+                    f"Found {len(member_emails)} members for group {group_name}"
+                )
+            else:
+                logger.debug(f"No members found for group {group_name}")
+
+    except Exception as e:
+        logger.error(f"Error building group member email map: {e}")
+
+    return group_member_emails
+
+
 def jira_group_sync(
     tenant_id: str,  # noqa: ARG001
     cc_pair: ConnectorCredentialPair,
 ) -> Generator[ExternalUserGroup, None, None]:
-    """Sync Jira groups and their members, yielding one group at a time.
+    """
+    Sync Jira groups and their members.
 
-    Streams group-by-group rather than accumulating all groups in memory.
+    This function fetches all groups from Jira and yields ExternalUserGroup
+    objects containing the group ID and member emails.
     """
     jira_base_url = cc_pair.connector.connector_specific_config.get("jira_base_url", "")
     scoped_token = cc_pair.connector.connector_specific_config.get(
@@ -127,26 +130,12 @@ def jira_group_sync(
         scoped_token=scoped_token,
     )
 
-    group_names = jira_client.groups()
-    if not group_names:
-        raise ValueError(f"No groups found for cc_pair_id={cc_pair.id}")
+    group_member_email_map = _build_group_member_email_map(jira_client=jira_client)
+    if not group_member_email_map:
+        raise ValueError(f"No groups with members found for cc_pair_id={cc_pair.id}")
 
-    logger.info(f"Found {len(group_names)} groups in Jira")
-
-    for group_name in group_names:
-        if not group_name:
-            continue
-
-        member_emails = _get_group_member_emails(
-            jira_client=jira_client,
-            group_name=group_name,
-        )
-        if not member_emails:
-            logger.debug(f"No members found for group {group_name}")
-            continue
-
-        logger.debug(f"Found {len(member_emails)} members for group {group_name}")
+    for group_id, group_member_emails in group_member_email_map.items():
         yield ExternalUserGroup(
-            id=group_name,
-            user_emails=list(member_emails),
+            id=group_id,
+            user_emails=list(group_member_emails),
         )

backend/ee/onyx/main.py

@@ -4,6 +4,7 @@ from contextlib import asynccontextmanager
 from fastapi import FastAPI
 from httpx_oauth.clients.google import GoogleOAuth2
 
+from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
 from ee.onyx.server.analytics.api import router as analytics_router
 from ee.onyx.server.auth_check import check_ee_router_auth
 from ee.onyx.server.billing.api import router as billing_router
@@ -152,9 +153,12 @@ def get_application() -> FastAPI:
     # License management
     include_router_with_global_prefix_prepended(application, license_router)
 
-    # Unified billing API - always registered in EE.
-    # Each endpoint is protected by the `current_admin_user` dependency (admin auth).
-    include_router_with_global_prefix_prepended(application, billing_router)
+    # Unified billing API - available when license system is enabled
+    # Works for both self-hosted and cloud deployments
+    # TODO(ENG-3533): Once frontend migrates to /admin/billing/*, this becomes the
+    # primary billing API and /tenants/* billing endpoints can be removed
+    if LICENSE_ENFORCEMENT_ENABLED:
+        include_router_with_global_prefix_prepended(application, billing_router)
 
     if MULTI_TENANT:
         # Tenant management

backend/ee/onyx/server/billing/api.py

@@ -26,6 +26,7 @@ import asyncio
 import httpx
 from fastapi import APIRouter
 from fastapi import Depends
+from fastapi import HTTPException
 from pydantic import BaseModel
 from sqlalchemy.orm import Session
 
@@ -41,6 +42,7 @@ from ee.onyx.server.billing.models import SeatUpdateRequest
 from ee.onyx.server.billing.models import SeatUpdateResponse
 from ee.onyx.server.billing.models import StripePublishableKeyResponse
 from ee.onyx.server.billing.models import SubscriptionStatusResponse
+from ee.onyx.server.billing.service import BillingServiceError
 from ee.onyx.server.billing.service import (
     create_checkout_session as create_checkout_service,
 )
@@ -56,8 +58,6 @@ from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_OVERRIDE
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_URL
 from onyx.configs.app_configs import WEB_DOMAIN
 from onyx.db.engine.sql_engine import get_session
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.redis.redis_pool import get_shared_redis_client
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
@@ -169,23 +169,26 @@ async def create_checkout_session(
     if seats is not None:
         used_seats = get_used_seats(tenant_id)
         if seats < used_seats:
-            raise OnyxError(
-                OnyxErrorCode.VALIDATION_ERROR,
-                f"Cannot subscribe with fewer seats than current usage. "
+            raise HTTPException(
+                status_code=400,
+                detail=f"Cannot subscribe with fewer seats than current usage. "
                 f"You have {used_seats} active users/integrations but requested {seats} seats.",
             )
 
     # Build redirect URL for after checkout completion
     redirect_url = f"{WEB_DOMAIN}/admin/billing?checkout=success"
 
-    return await create_checkout_service(
-        billing_period=billing_period,
-        seats=seats,
-        email=email,
-        license_data=license_data,
-        redirect_url=redirect_url,
-        tenant_id=tenant_id,
-    )
+    try:
+        return await create_checkout_service(
+            billing_period=billing_period,
+            seats=seats,
+            email=email,
+            license_data=license_data,
+            redirect_url=redirect_url,
+            tenant_id=tenant_id,
+        )
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)
 
 
 @router.post("/create-customer-portal-session")
@@ -203,15 +206,18 @@ async def create_customer_portal_session(
 
     # Self-hosted requires license
     if not MULTI_TENANT and not license_data:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "No license found")
+        raise HTTPException(status_code=400, detail="No license found")
 
     return_url = request.return_url if request else f"{WEB_DOMAIN}/admin/billing"
 
-    return await create_portal_service(
-        license_data=license_data,
-        return_url=return_url,
-        tenant_id=tenant_id,
-    )
+    try:
+        return await create_portal_service(
+            license_data=license_data,
+            return_url=return_url,
+            tenant_id=tenant_id,
+        )
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)
 
 
 @router.get("/billing-information")
@@ -234,9 +240,9 @@ async def get_billing_information(
 
     # Check circuit breaker (self-hosted only)
     if _is_billing_circuit_open():
-        raise OnyxError(
-            OnyxErrorCode.SERVICE_UNAVAILABLE,
-            "Stripe connection temporarily disabled. Click 'Connect to Stripe' to retry.",
+        raise HTTPException(
+            status_code=503,
+            detail="Stripe connection temporarily disabled. Click 'Connect to Stripe' to retry.",
         )
 
     try:
@@ -244,15 +250,11 @@ async def get_billing_information(
             license_data=license_data,
             tenant_id=tenant_id,
         )
-    except OnyxError as e:
+    except BillingServiceError as e:
         # Open circuit breaker on connection failures (self-hosted only)
-        if e.status_code in (
-            OnyxErrorCode.BAD_GATEWAY.status_code,
-            OnyxErrorCode.SERVICE_UNAVAILABLE.status_code,
-            OnyxErrorCode.GATEWAY_TIMEOUT.status_code,
-        ):
+        if e.status_code in (502, 503, 504):
             _open_billing_circuit()
-        raise
+        raise HTTPException(status_code=e.status_code, detail=e.message)
 
 
 @router.post("/seats/update")
@@ -272,25 +274,31 @@ async def update_seats(
 
     # Self-hosted requires license
     if not MULTI_TENANT and not license_data:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "No license found")
+        raise HTTPException(status_code=400, detail="No license found")
 
     # Validate that new seat count is not less than current used seats
     used_seats = get_used_seats(tenant_id)
     if request.new_seat_count < used_seats:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            f"Cannot reduce seats below current usage. "
+        raise HTTPException(
+            status_code=400,
+            detail=f"Cannot reduce seats below current usage. "
             f"You have {used_seats} active users/integrations but requested {request.new_seat_count} seats.",
         )
 
-    # Note: Don't store license here - the control plane may still be processing
-    # the subscription update. The frontend should call /license/claim after a
-    # short delay to get the freshly generated license.
-    return await update_seat_service(
-        new_seat_count=request.new_seat_count,
-        license_data=license_data,
-        tenant_id=tenant_id,
-    )
+    try:
+        result = await update_seat_service(
+            new_seat_count=request.new_seat_count,
+            license_data=license_data,
+            tenant_id=tenant_id,
+        )
+
+        # Note: Don't store license here - the control plane may still be processing
+        # the subscription update. The frontend should call /license/claim after a
+        # short delay to get the freshly generated license.
+
+        return result
+    except BillingServiceError as e:
+        raise HTTPException(status_code=e.status_code, detail=e.message)
 
 
 @router.get("/stripe-publishable-key")
@@ -321,18 +329,18 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
         if STRIPE_PUBLISHABLE_KEY_OVERRIDE:
             key = STRIPE_PUBLISHABLE_KEY_OVERRIDE.strip()
             if not key.startswith("pk_"):
-                raise OnyxError(
-                    OnyxErrorCode.INTERNAL_ERROR,
-                    "Invalid Stripe publishable key format",
+                raise HTTPException(
+                    status_code=500,
+                    detail="Invalid Stripe publishable key format",
                 )
             _stripe_publishable_key_cache = key
             return StripePublishableKeyResponse(publishable_key=key)
 
         # Fall back to S3 bucket
         if not STRIPE_PUBLISHABLE_KEY_URL:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Stripe publishable key is not configured",
+            raise HTTPException(
+                status_code=500,
+                detail="Stripe publishable key is not configured",
             )
 
         try:
@@ -343,17 +351,17 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
 
                 # Validate key format
                 if not key.startswith("pk_"):
-                    raise OnyxError(
-                        OnyxErrorCode.INTERNAL_ERROR,
-                        "Invalid Stripe publishable key format",
+                    raise HTTPException(
+                        status_code=500,
+                        detail="Invalid Stripe publishable key format",
                     )
 
                 _stripe_publishable_key_cache = key
                 return StripePublishableKeyResponse(publishable_key=key)
         except httpx.HTTPError:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Failed to fetch Stripe publishable key",
+            raise HTTPException(
+                status_code=500,
+                detail="Failed to fetch Stripe publishable key",
             )
 
 

backend/ee/onyx/server/billing/service.py

@@ -22,8 +22,6 @@ from ee.onyx.server.billing.models import SeatUpdateResponse
 from ee.onyx.server.billing.models import SubscriptionStatusResponse
 from ee.onyx.server.tenants.access import generate_data_plane_token
 from onyx.configs.app_configs import CONTROL_PLANE_API_BASE_URL
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 
@@ -33,6 +31,15 @@ logger = setup_logger()
 _REQUEST_TIMEOUT = 30.0
 
 
+class BillingServiceError(Exception):
+    """Exception raised for billing service errors."""
+
+    def __init__(self, message: str, status_code: int = 500):
+        self.message = message
+        self.status_code = status_code
+        super().__init__(self.message)
+
+
 def _get_proxy_headers(license_data: str | None) -> dict[str, str]:
     """Build headers for proxy requests (self-hosted).
 
@@ -94,7 +101,7 @@ async def _make_billing_request(
         Response JSON as dict
 
     Raises:
-        OnyxError: If request fails
+        BillingServiceError: If request fails
     """
 
     base_url = _get_base_url()
@@ -121,17 +128,11 @@ async def _make_billing_request(
         except Exception:
             pass
         logger.error(f"{error_message}: {e.response.status_code} - {detail}")
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY,
-            detail,
-            status_code_override=e.response.status_code,
-        )
+        raise BillingServiceError(detail, e.response.status_code)
 
     except httpx.RequestError:
         logger.exception("Failed to connect to billing service")
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, "Failed to connect to billing service"
-        )
+        raise BillingServiceError("Failed to connect to billing service", 502)
 
 
 async def create_checkout_session(

backend/ee/onyx/server/enterprise_settings/api.py

@@ -223,15 +223,6 @@ def get_active_scim_token(
     token = dal.get_active_token()
     if not token:
         raise HTTPException(status_code=404, detail="No active SCIM token")
-
-    # Derive the IdP domain from the first synced user as a heuristic.
-    idp_domain: str | None = None
-    mappings, _total = dal.list_user_mappings(start_index=1, count=1)
-    if mappings:
-        user = dal.get_user(mappings[0].user_id)
-        if user and "@" in user.email:
-            idp_domain = user.email.rsplit("@", 1)[1]
-
     return ScimTokenResponse(
         id=token.id,
         name=token.name,
@@ -239,7 +230,6 @@ def get_active_scim_token(
         is_active=token.is_active,
         created_at=token.created_at,
         last_used_at=token.last_used_at,
-        idp_domain=idp_domain,
     )
 
 

backend/ee/onyx/server/license/api.py

@@ -14,6 +14,7 @@ import requests
 from fastapi import APIRouter
 from fastapi import Depends
 from fastapi import File
+from fastapi import HTTPException
 from fastapi import UploadFile
 from sqlalchemy.orm import Session
 
@@ -34,8 +35,6 @@ from ee.onyx.server.license.models import SeatUsageResponse
 from ee.onyx.utils.license import verify_license_signature
 from onyx.auth.users import User
 from onyx.db.engine.sql_engine import get_session
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 
@@ -128,9 +127,9 @@ async def claim_license(
     2. Without session_id: Re-claim using existing license for auth
     """
     if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License claiming is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License claiming is only available for self-hosted deployments",
         )
 
     try:
@@ -147,16 +146,15 @@ async def claim_license(
             # Re-claim using existing license for auth
             metadata = get_license_metadata(db_session)
             if not metadata or not metadata.tenant_id:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    "No license found. Provide session_id after checkout.",
+                raise HTTPException(
+                    status_code=400,
+                    detail="No license found. Provide session_id after checkout.",
                 )
 
             license_row = get_license(db_session)
             if not license_row or not license_row.license_data:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    "No license found in database",
+                raise HTTPException(
+                    status_code=400, detail="No license found in database"
                 )
 
             url = f"{CLOUD_DATA_PLANE_URL}/proxy/license/{metadata.tenant_id}"
@@ -175,7 +173,7 @@ async def claim_license(
         license_data = data.get("license")
 
         if not license_data:
-            raise OnyxError(OnyxErrorCode.NOT_FOUND, "No license in response")
+            raise HTTPException(status_code=404, detail="No license in response")
 
         # Verify signature before persisting
         payload = verify_license_signature(license_data)
@@ -201,14 +199,12 @@ async def claim_license(
             detail = error_data.get("detail", detail)
         except Exception:
             pass
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=status_code
-        )
+        raise HTTPException(status_code=status_code, detail=detail)
     except ValueError as e:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, str(e))
+        raise HTTPException(status_code=400, detail=str(e))
     except requests.RequestException:
-        raise OnyxError(
-            OnyxErrorCode.BAD_GATEWAY, "Failed to connect to license server"
+        raise HTTPException(
+            status_code=502, detail="Failed to connect to license server"
         )
 
 
@@ -225,9 +221,9 @@ async def upload_license(
     The license file must be cryptographically signed by Onyx.
     """
     if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License upload is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License upload is only available for self-hosted deployments",
         )
 
     try:
@@ -238,14 +234,14 @@ async def upload_license(
         # Remove any stray whitespace/newlines from user input
         license_data = license_data.strip()
     except UnicodeDecodeError:
-        raise OnyxError(OnyxErrorCode.INVALID_INPUT, "Invalid license file format")
+        raise HTTPException(status_code=400, detail="Invalid license file format")
 
     # Verify cryptographic signature - this is the only validation needed
     # The license's tenant_id identifies the customer in control plane, not locally
     try:
         payload = verify_license_signature(license_data)
     except ValueError as e:
-        raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, str(e))
+        raise HTTPException(status_code=400, detail=str(e))
 
     # Persist to DB and update cache
     upsert_license(db_session, license_data)
@@ -301,9 +297,9 @@ async def delete_license(
     Admin only - removes license from database and invalidates cache.
     """
     if MULTI_TENANT:
-        raise OnyxError(
-            OnyxErrorCode.VALIDATION_ERROR,
-            "License deletion is only available for self-hosted deployments",
+        raise HTTPException(
+            status_code=400,
+            detail="License deletion is only available for self-hosted deployments",
         )
 
     try:

backend/ee/onyx/server/middleware/license_enforcement.py

@@ -46,6 +46,7 @@ from fastapi import FastAPI
 from fastapi import Request
 from fastapi import Response
 from fastapi.responses import JSONResponse
+from redis.exceptions import RedisError
 from sqlalchemy.exc import SQLAlchemyError
 
 from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
@@ -55,7 +56,6 @@ from ee.onyx.configs.license_enforcement_config import (
 )
 from ee.onyx.db.license import get_cached_license_metadata
 from ee.onyx.db.license import refresh_license_cache
-from onyx.cache.interface import CACHE_TRANSIENT_ERRORS
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.server.settings.models import ApplicationStatus
 from shared_configs.contextvars import get_current_tenant_id
@@ -164,9 +164,9 @@ def add_license_enforcement_middleware(
                     "[license_enforcement] No license, allowing community features"
                 )
                 is_gated = False
-        except CACHE_TRANSIENT_ERRORS as e:
+        except RedisError as e:
             logger.warning(f"Failed to check license metadata: {e}")
-            # Fail open - don't block users due to cache connectivity issues
+            # Fail open - don't block users due to Redis connectivity issues
             is_gated = False
 
         if is_gated:

backend/ee/onyx/server/scim/api.py

@@ -423,63 +423,15 @@ def create_user(
 
     email = user_resource.userName.strip()
 
-    # Check for existing user — if they exist but aren't SCIM-managed yet,
-    # link them to the IdP rather than rejecting with 409.
-    external_id: str | None = user_resource.externalId
-    scim_username: str = user_resource.userName.strip()
-    fields: ScimMappingFields = _fields_from_resource(user_resource)
-
-    existing_user = dal.get_user_by_email(email)
-    if existing_user:
-        existing_mapping = dal.get_user_mapping_by_user_id(existing_user.id)
-        if existing_mapping:
-            return _scim_error_response(409, f"User with email {email} already exists")
-
-        # Adopt pre-existing user into SCIM management.
-        # Reactivating a deactivated user consumes a seat, so enforce the
-        # seat limit the same way replace_user does.
-        if user_resource.active and not existing_user.is_active:
-            seat_error = _check_seat_availability(dal)
-            if seat_error:
-                return _scim_error_response(403, seat_error)
-
-        personal_name = _scim_name_to_str(user_resource.name)
-        dal.update_user(
-            existing_user,
-            is_active=user_resource.active,
-            **({"personal_name": personal_name} if personal_name else {}),
-        )
-
-        try:
-            dal.create_user_mapping(
-                external_id=external_id,
-                user_id=existing_user.id,
-                scim_username=scim_username,
-                fields=fields,
-            )
-            dal.commit()
-        except IntegrityError:
-            dal.rollback()
-            return _scim_error_response(
-                409, f"User with email {email} already has a SCIM mapping"
-            )
-
-        return _scim_resource_response(
-            provider.build_user_resource(
-                existing_user,
-                external_id,
-                scim_username=scim_username,
-                fields=fields,
-            ),
-            status_code=201,
-        )
-
-    # Only enforce seat limit for net-new users — adopting a pre-existing
-    # user doesn't consume a new seat.
+    # Enforce seat limit
     seat_error = _check_seat_availability(dal)
     if seat_error:
         return _scim_error_response(403, seat_error)
 
+    # Check for existing user
+    if dal.get_user_by_email(email):
+        return _scim_error_response(409, f"User with email {email} already exists")
+
     # Create user with a random password (SCIM users authenticate via IdP)
     personal_name = _scim_name_to_str(user_resource.name)
     user = User(
@@ -497,21 +449,21 @@ def create_user(
         dal.rollback()
         return _scim_error_response(409, f"User with email {email} already exists")
 
-    # Always create a SCIM mapping so that the user is marked as
-    # SCIM-managed. externalId may be None (RFC 7643 says it's optional).
-    try:
+    # Create SCIM mapping when externalId is provided — this is how the IdP
+    # correlates this user on subsequent requests.  Per RFC 7643, externalId
+    # is optional and assigned by the provisioning client.
+    external_id = user_resource.externalId
+    scim_username = user_resource.userName.strip()
+    fields = _fields_from_resource(user_resource)
+    if external_id:
         dal.create_user_mapping(
             external_id=external_id,
             user_id=user.id,
             scim_username=scim_username,
             fields=fields,
         )
-        dal.commit()
-    except IntegrityError:
-        dal.rollback()
-        return _scim_error_response(
-            409, f"User with email {email} already has a SCIM mapping"
-        )
+
+    dal.commit()
 
     return _scim_resource_response(
         provider.build_user_resource(

backend/ee/onyx/server/scim/models.py

@@ -365,7 +365,6 @@ class ScimTokenResponse(BaseModel):
     is_active: bool
     created_at: datetime
     last_used_at: datetime | None = None
-    idp_domain: str | None = None
 
 
 class ScimTokenCreatedResponse(ScimTokenResponse):

backend/ee/onyx/server/scim/providers/base.py

@@ -170,10 +170,7 @@ class ScimProvider(ABC):
                 formatted=user.personal_name or "",
             )
         if not user.personal_name:
-            # Derive a reasonable name from the email so that SCIM spec tests
-            # see non-empty givenName / familyName for every user resource.
-            local = user.email.split("@")[0] if user.email else ""
-            return ScimName(givenName=local, familyName="", formatted=local)
+            return ScimName(givenName="", familyName="", formatted="")
         parts = user.personal_name.split(" ", 1)
         return ScimName(
             givenName=parts[0],

backend/ee/onyx/server/seeding.py

@@ -26,7 +26,6 @@ from onyx.db.models import Tool
 from onyx.db.persona import upsert_persona
 from onyx.server.features.persona.models import PersonaUpsertRequest
 from onyx.server.manage.llm.models import LLMProviderUpsertRequest
-from onyx.server.manage.llm.models import LLMProviderView
 from onyx.server.settings.models import Settings
 from onyx.server.settings.store import store_settings as store_base_settings
 from onyx.utils.logger import setup_logger
@@ -126,16 +125,10 @@ def _seed_llms(
         existing = fetch_existing_llm_provider(name=request.name, db_session=db_session)
         if existing:
             request.id = existing.id
-    seeded_providers: list[LLMProviderView] = []
-    for llm_upsert_request in llm_upsert_requests:
-        try:
-            seeded_providers.append(upsert_llm_provider(llm_upsert_request, db_session))
-        except ValueError as e:
-            logger.warning(
-                "Failed to upsert LLM provider '%s' during seeding: %s",
-                llm_upsert_request.name,
-                e,
-            )
+    seeded_providers = [
+        upsert_llm_provider(llm_upsert_request, db_session)
+        for llm_upsert_request in llm_upsert_requests
+    ]
 
     default_provider = next(
         (p for p in seeded_providers if p.model_configurations), None

backend/ee/onyx/server/settings/api.py

@@ -6,7 +6,6 @@ from sqlalchemy.exc import SQLAlchemyError
 from ee.onyx.configs.app_configs import LICENSE_ENFORCEMENT_ENABLED
 from ee.onyx.db.license import get_cached_license_metadata
 from ee.onyx.db.license import refresh_license_cache
-from onyx.cache.interface import CACHE_TRANSIENT_ERRORS
 from onyx.configs.app_configs import ENTERPRISE_EDITION_ENABLED
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.server.settings.models import ApplicationStatus
@@ -126,7 +125,7 @@ def apply_license_status_to_settings(settings: Settings) -> Settings:
                 # syncing) means indexed data may need protection.
                 settings.application_status = _BLOCKING_STATUS
             settings.ee_features_enabled = False
-    except CACHE_TRANSIENT_ERRORS as e:
+    except RedisError as e:
         logger.warning(f"Failed to check license metadata for settings: {e}")
         # Fail closed - disable EE features if we can't verify license
         settings.ee_features_enabled = False

backend/ee/onyx/server/tenants/billing_api.py

@@ -21,6 +21,7 @@ import asyncio
 import httpx
 from fastapi import APIRouter
 from fastapi import Depends
+from fastapi import HTTPException
 
 from ee.onyx.auth.users import current_admin_user
 from ee.onyx.server.tenants.access import control_plane_dep
@@ -42,8 +43,6 @@ from onyx.auth.users import User
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_OVERRIDE
 from onyx.configs.app_configs import STRIPE_PUBLISHABLE_KEY_URL
 from onyx.configs.app_configs import WEB_DOMAIN
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
 from onyx.utils.logger import setup_logger
 from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
 from shared_configs.contextvars import get_current_tenant_id
@@ -117,14 +116,9 @@ async def create_customer_portal_session(
     try:
         portal_url = fetch_customer_portal_session(tenant_id, return_url)
         return {"stripe_customer_portal_url": portal_url}
-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
         logger.exception("Failed to create customer portal session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create customer portal session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))
 
 
 @router.post("/create-checkout-session")
@@ -140,14 +134,9 @@ async def create_checkout_session(
     try:
         checkout_url = fetch_stripe_checkout_session(tenant_id, billing_period, seats)
         return {"stripe_checkout_url": checkout_url}
-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
         logger.exception("Failed to create checkout session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create checkout session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))
 
 
 @router.post("/create-subscription-session")
@@ -158,20 +147,15 @@ async def create_subscription_session(
     try:
         tenant_id = CURRENT_TENANT_ID_CONTEXTVAR.get()
         if not tenant_id:
-            raise OnyxError(OnyxErrorCode.VALIDATION_ERROR, "Tenant ID not found")
+            raise HTTPException(status_code=400, detail="Tenant ID not found")
 
         billing_period = request.billing_period if request else "monthly"
         session_id = fetch_stripe_checkout_session(tenant_id, billing_period)
         return SubscriptionSessionResponse(sessionId=session_id)
 
-    except OnyxError:
-        raise
-    except Exception:
+    except Exception as e:
         logger.exception("Failed to create subscription session")
-        raise OnyxError(
-            OnyxErrorCode.INTERNAL_ERROR,
-            "Failed to create subscription session",
-        )
+        raise HTTPException(status_code=500, detail=str(e))
 
 
 @router.get("/stripe-publishable-key")
@@ -202,18 +186,18 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
         if STRIPE_PUBLISHABLE_KEY_OVERRIDE:
             key = STRIPE_PUBLISHABLE_KEY_OVERRIDE.strip()
             if not key.startswith("pk_"):
-                raise OnyxError(
-                    OnyxErrorCode.INTERNAL_ERROR,
-                    "Invalid Stripe publishable key format",
+                raise HTTPException(
+                    status_code=500,
+                    detail="Invalid Stripe publishable key format",
                 )
             _stripe_publishable_key_cache = key
             return StripePublishableKeyResponse(publishable_key=key)
 
         # Fall back to S3 bucket
         if not STRIPE_PUBLISHABLE_KEY_URL:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Stripe publishable key is not configured",
+            raise HTTPException(
+                status_code=500,
+                detail="Stripe publishable key is not configured",
             )
 
         try:
@@ -224,15 +208,15 @@ async def get_stripe_publishable_key() -> StripePublishableKeyResponse:
 
                 # Validate key format
                 if not key.startswith("pk_"):
-                    raise OnyxError(
-                        OnyxErrorCode.INTERNAL_ERROR,
-                        "Invalid Stripe publishable key format",
+                    raise HTTPException(
+                        status_code=500,
+                        detail="Invalid Stripe publishable key format",
                     )
 
                 _stripe_publishable_key_cache = key
                 return StripePublishableKeyResponse(publishable_key=key)
         except httpx.HTTPError:
-            raise OnyxError(
-                OnyxErrorCode.INTERNAL_ERROR,
-                "Failed to fetch Stripe publishable key",
+            raise HTTPException(
+                status_code=500,
+                detail="Failed to fetch Stripe publishable key",
             )

backend/ee/onyx/server/user_group/api.py

@@ -5,8 +5,6 @@ from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session
 
 from ee.onyx.db.user_group import add_users_to_user_group
-from ee.onyx.db.user_group import delete_user_group as db_delete_user_group
-from ee.onyx.db.user_group import fetch_user_group
 from ee.onyx.db.user_group import fetch_user_groups
 from ee.onyx.db.user_group import fetch_user_groups_for_user
 from ee.onyx.db.user_group import insert_user_group
@@ -22,7 +20,6 @@ from ee.onyx.server.user_group.models import UserGroupUpdate
 from onyx.auth.users import current_admin_user
 from onyx.auth.users import current_curator_or_admin_user
 from onyx.auth.users import current_user
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.configs.constants import PUBLIC_API_TAGS
 from onyx.db.engine.sql_engine import get_session
 from onyx.db.models import User
@@ -156,8 +153,3 @@ def delete_user_group(
         prepare_user_group_for_deletion(db_session, user_group_id)
     except ValueError as e:
         raise HTTPException(status_code=404, detail=str(e))
-
-    if DISABLE_VECTOR_DB:
-        user_group = fetch_user_group(db_session, user_group_id)
-        if user_group:
-            db_delete_user_group(db_session, user_group)

backend/ee/onyx/utils/encryption.py

@@ -14,91 +14,67 @@ from onyx.utils.variable_functionality import fetch_versioned_implementation
 logger = setup_logger()
 
 
-@lru_cache(maxsize=2)
+@lru_cache(maxsize=1)
 def _get_trimmed_key(key: str) -> bytes:
     encoded_key = key.encode()
     key_length = len(encoded_key)
     if key_length < 16:
         raise RuntimeError("Invalid ENCRYPTION_KEY_SECRET - too short")
+    elif key_length > 32:
+        key = key[:32]
+    elif key_length not in (16, 24, 32):
+        valid_lengths = [16, 24, 32]
+        key = key[: min(valid_lengths, key=lambda x: abs(x - key_length))]
 
-    # Trim to the largest valid AES key size that fits
-    valid_lengths = [32, 24, 16]
-    for size in valid_lengths:
-        if key_length >= size:
-            return encoded_key[:size]
-
-    raise AssertionError("unreachable")
+    return encoded_key
 
 
-def _encrypt_string(input_str: str, key: str | None = None) -> bytes:
-    effective_key = key if key is not None else ENCRYPTION_KEY_SECRET
-    if not effective_key:
+def _encrypt_string(input_str: str) -> bytes:
+    if not ENCRYPTION_KEY_SECRET:
         return input_str.encode()
 
-    trimmed = _get_trimmed_key(effective_key)
+    key = _get_trimmed_key(ENCRYPTION_KEY_SECRET)
     iv = urandom(16)
     padder = padding.PKCS7(algorithms.AES.block_size).padder()
     padded_data = padder.update(input_str.encode()) + padder.finalize()
 
-    cipher = Cipher(algorithms.AES(trimmed), modes.CBC(iv), backend=default_backend())
+    cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
     encryptor = cipher.encryptor()
     encrypted_data = encryptor.update(padded_data) + encryptor.finalize()
 
     return iv + encrypted_data
 
 
-def _decrypt_bytes(input_bytes: bytes, key: str | None = None) -> str:
-    effective_key = key if key is not None else ENCRYPTION_KEY_SECRET
-    if not effective_key:
+def _decrypt_bytes(input_bytes: bytes) -> str:
+    if not ENCRYPTION_KEY_SECRET:
         return input_bytes.decode()
 
-    trimmed = _get_trimmed_key(effective_key)
-    try:
-        iv = input_bytes[:16]
-        encrypted_data = input_bytes[16:]
+    key = _get_trimmed_key(ENCRYPTION_KEY_SECRET)
+    iv = input_bytes[:16]
+    encrypted_data = input_bytes[16:]
 
-        cipher = Cipher(
-            algorithms.AES(trimmed), modes.CBC(iv), backend=default_backend()
-        )
-        decryptor = cipher.decryptor()
-        decrypted_padded_data = decryptor.update(encrypted_data) + decryptor.finalize()
+    cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
+    decryptor = cipher.decryptor()
+    decrypted_padded_data = decryptor.update(encrypted_data) + decryptor.finalize()
 
-        unpadder = padding.PKCS7(algorithms.AES.block_size).unpadder()
-        decrypted_data = unpadder.update(decrypted_padded_data) + unpadder.finalize()
+    unpadder = padding.PKCS7(algorithms.AES.block_size).unpadder()
+    decrypted_data = unpadder.update(decrypted_padded_data) + unpadder.finalize()
 
-        return decrypted_data.decode()
-    except (ValueError, UnicodeDecodeError):
-        if key is not None:
-            # Explicit key was provided — don't fall back silently
-            raise
-        # Read path: attempt raw UTF-8 decode as a fallback for legacy data.
-        # Does NOT handle data encrypted with a different key — that
-        # ciphertext is not valid UTF-8 and will raise below.
-        logger.warning(
-            "AES decryption failed — falling back to raw decode. "
-            "Run the re-encrypt secrets script to rotate to the current key."
-        )
-        try:
-            return input_bytes.decode()
-        except UnicodeDecodeError:
-            raise ValueError(
-                "Data is not valid UTF-8 — likely encrypted with a different key. "
-                "Run the re-encrypt secrets script to rotate to the current key."
-            ) from None
+    return decrypted_data.decode()
 
 
-def encrypt_string_to_bytes(input_str: str, key: str | None = None) -> bytes:
+def encrypt_string_to_bytes(input_str: str) -> bytes:
     versioned_encryption_fn = fetch_versioned_implementation(
         "onyx.utils.encryption", "_encrypt_string"
     )
-    return versioned_encryption_fn(input_str, key=key)
+    return versioned_encryption_fn(input_str)
 
 
-def decrypt_bytes_to_string(input_bytes: bytes, key: str | None = None) -> str:
+def decrypt_bytes_to_string(input_bytes: bytes) -> str:
     versioned_decryption_fn = fetch_versioned_implementation(
         "onyx.utils.encryption", "_decrypt_bytes"
     )
-    return versioned_decryption_fn(input_bytes, key=key)
+    return versioned_decryption_fn(input_bytes)
 
 
 def test_encryption() -> None:

backend/onyx/access/access.py

@@ -1,6 +1,7 @@
 from collections.abc import Callable
 from typing import cast
 
+from sqlalchemy.orm import joinedload
 from sqlalchemy.orm import Session
 
 from onyx.access.models import DocumentAccess
@@ -11,7 +12,6 @@ from onyx.db.document import get_access_info_for_document
 from onyx.db.document import get_access_info_for_documents
 from onyx.db.models import User
 from onyx.db.models import UserFile
-from onyx.db.user_file import fetch_user_files_with_access_relationships
 from onyx.utils.variable_functionality import fetch_ee_implementation_or_noop
 from onyx.utils.variable_functionality import fetch_versioned_implementation
 
@@ -132,61 +132,19 @@ def get_access_for_user_files(
     user_file_ids: list[str],
     db_session: Session,
 ) -> dict[str, DocumentAccess]:
-    versioned_fn = fetch_versioned_implementation(
-        "onyx.access.access", "get_access_for_user_files_impl"
+    user_files = (
+        db_session.query(UserFile)
+        .options(joinedload(UserFile.user))  # Eager load the user relationship
+        .filter(UserFile.id.in_(user_file_ids))
+        .all()
     )
-    return versioned_fn(user_file_ids, db_session)
-
-
-def get_access_for_user_files_impl(
-    user_file_ids: list[str],
-    db_session: Session,
-) -> dict[str, DocumentAccess]:
-    user_files = fetch_user_files_with_access_relationships(user_file_ids, db_session)
-    return build_access_for_user_files_impl(user_files)
-
-
-def build_access_for_user_files(
-    user_files: list[UserFile],
-) -> dict[str, DocumentAccess]:
-    """Compute access from pre-loaded UserFile objects (with relationships).
-    Callers must ensure UserFile.user, Persona.users, and Persona.user are
-    eagerly loaded (and Persona.groups for the EE path)."""
-    versioned_fn = fetch_versioned_implementation(
-        "onyx.access.access", "build_access_for_user_files_impl"
-    )
-    return versioned_fn(user_files)
-
-
-def build_access_for_user_files_impl(
-    user_files: list[UserFile],
-) -> dict[str, DocumentAccess]:
-    result: dict[str, DocumentAccess] = {}
-    for user_file in user_files:
-        emails, is_public = collect_user_file_access(user_file)
-        result[str(user_file.id)] = DocumentAccess.build(
-            user_emails=list(emails),
+    return {
+        str(user_file.id): DocumentAccess.build(
+            user_emails=[user_file.user.email] if user_file.user else [],
             user_groups=[],
-            is_public=is_public,
+            is_public=True if user_file.user is None else False,
             external_user_emails=[],
             external_user_group_ids=[],
         )
-    return result
-
-
-def collect_user_file_access(user_file: UserFile) -> tuple[set[str], bool]:
-    """Collect all user emails that should have access to this user file.
-    Includes the owner plus any users who have access via shared personas.
-    Returns (emails, is_public)."""
-    emails: set[str] = {user_file.user.email}
-    is_public = False
-    for persona in user_file.assistants:
-        if persona.deleted:
-            continue
-        if persona.is_public:
-            is_public = True
-        if persona.user_id is not None and persona.user:
-            emails.add(persona.user.email)
-        for shared_user in persona.users:
-            emails.add(shared_user.email)
-    return emails, is_public
+        for user_file in user_files
+    }

backend/onyx/auth/users.py

@@ -1,7 +1,4 @@
-import base64
-import hashlib
 import json
-import os
 import random
 import secrets
 import string
@@ -31,8 +28,6 @@ from fastapi import Query
 from fastapi import Request
 from fastapi import Response
 from fastapi import status
-from fastapi import WebSocket
-from fastapi.responses import JSONResponse
 from fastapi.responses import RedirectResponse
 from fastapi.security import OAuth2PasswordRequestForm
 from fastapi_users import BaseUserManager
@@ -59,7 +54,6 @@ from fastapi_users.router.common import ErrorModel
 from fastapi_users_db_sqlalchemy import SQLAlchemyUserDatabase
 from httpx_oauth.integrations.fastapi import OAuth2AuthorizeCallback
 from httpx_oauth.oauth2 import BaseOAuth2
-from httpx_oauth.oauth2 import GetAccessTokenError
 from httpx_oauth.oauth2 import OAuth2Token
 from pydantic import BaseModel
 from sqlalchemy import nulls_last
@@ -125,12 +119,8 @@ from onyx.db.models import Persona
 from onyx.db.models import User
 from onyx.db.pat import fetch_user_for_pat
 from onyx.db.users import get_user_by_email
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import log_onyx_error
-from onyx.error_handling.exceptions import onyx_error_to_json_response
-from onyx.error_handling.exceptions import OnyxError
 from onyx.redis.redis_pool import get_async_redis_connection
-from onyx.redis.redis_pool import retrieve_ws_token_data
+from onyx.redis.redis_pool import get_redis_client
 from onyx.server.settings.store import load_settings
 from onyx.server.utils import BasicAuthenticationError
 from onyx.utils.logger import setup_logger
@@ -156,22 +146,10 @@ def is_user_admin(user: User) -> bool:
 
 
 def verify_auth_setting() -> None:
-    """Log warnings for AUTH_TYPE issues.
-
-    This only runs on app startup not during migrations/scripts.
-    """
-    raw_auth_type = (os.environ.get("AUTH_TYPE") or "").lower()
-
-    if raw_auth_type == "cloud":
+    if AUTH_TYPE == AuthType.CLOUD:
         raise ValueError(
-            "'cloud' is not a valid auth type for self-hosted deployments."
+            f"{AUTH_TYPE.value} is not a valid auth type for self-hosted deployments."
         )
-    if raw_auth_type == "disabled":
-        logger.warning(
-            "AUTH_TYPE='disabled' is no longer supported. "
-            "Using 'basic' instead. Please update your configuration."
-        )
-
     logger.notice(f"Using Auth Type: {AUTH_TYPE.value}")
 
 
@@ -223,14 +201,13 @@ def user_needs_to_be_verified() -> bool:
 
 
 def anonymous_user_enabled(*, tenant_id: str | None = None) -> bool:
-    from onyx.cache.factory import get_cache_backend
-
-    cache = get_cache_backend(tenant_id=tenant_id)
-    value = cache.get(OnyxRedisLocks.ANONYMOUS_USER_ENABLED)
+    redis_client = get_redis_client(tenant_id=tenant_id)
+    value = redis_client.get(OnyxRedisLocks.ANONYMOUS_USER_ENABLED)
 
     if value is None:
         return False
 
+    assert isinstance(value, bytes)
     return int(value.decode("utf-8")) == 1
 
 
@@ -1622,102 +1599,6 @@ async def current_admin_user(user: User = Depends(current_user)) -> User:
     return user
 
 
-async def _get_user_from_token_data(token_data: dict) -> User | None:
-    """Shared logic: token data dict → User object.
-
-    Args:
-        token_data: Decoded token data containing 'sub' (user ID).
-
-    Returns:
-        User object if found and active, None otherwise.
-    """
-    user_id = token_data.get("sub")
-    if not user_id:
-        return None
-
-    try:
-        user_uuid = uuid.UUID(user_id)
-    except ValueError:
-        return None
-
-    async with get_async_session_context_manager() as async_db_session:
-        user = await async_db_session.get(User, user_uuid)
-        if user is None or not user.is_active:
-            return None
-        return user
-
-
-async def current_user_from_websocket(
-    websocket: WebSocket,
-    token: str = Query(..., description="WebSocket authentication token"),
-) -> User:
-    """
-    WebSocket authentication dependency using query parameter.
-
-    Validates the WS token from query param and returns the User.
-    Raises BasicAuthenticationError if authentication fails.
-
-    The token must be obtained from POST /voice/ws-token before connecting.
-    Tokens are single-use and expire after 60 seconds.
-
-    Usage:
-        1. POST /voice/ws-token -> {"token": "xxx"}
-        2. Connect to ws://host/path?token=xxx
-
-    This applies the same auth checks as current_user() for HTTP endpoints.
-    """
-    # Check Origin header to prevent Cross-Site WebSocket Hijacking (CSWSH)
-    # Browsers always send Origin on WebSocket connections
-    origin = websocket.headers.get("origin")
-    expected_origin = WEB_DOMAIN.rstrip("/")
-    if not origin:
-        logger.warning("WS auth: missing Origin header")
-        raise BasicAuthenticationError(detail="Access denied. Missing origin.")
-
-    actual_origin = origin.rstrip("/")
-    if actual_origin != expected_origin:
-        logger.warning(
-            f"WS auth: origin mismatch. Expected {expected_origin}, got {actual_origin}"
-        )
-        raise BasicAuthenticationError(detail="Access denied. Invalid origin.")
-
-    # Validate WS token in Redis (single-use, deleted after retrieval)
-    try:
-        token_data = await retrieve_ws_token_data(token)
-        if token_data is None:
-            raise BasicAuthenticationError(
-                detail="Access denied. Invalid or expired authentication token."
-            )
-    except BasicAuthenticationError:
-        raise
-    except Exception as e:
-        logger.error(f"WS auth: error during token validation: {e}")
-        raise BasicAuthenticationError(
-            detail="Authentication verification failed."
-        ) from e
-
-    # Get user from token data
-    user = await _get_user_from_token_data(token_data)
-    if user is None:
-        logger.warning(f"WS auth: user not found for id={token_data.get('sub')}")
-        raise BasicAuthenticationError(
-            detail="Access denied. User not found or inactive."
-        )
-
-    # Apply same checks as HTTP auth (verification, OIDC expiry, role)
-    user = await double_check_user(user)
-
-    # Block LIMITED users (same as current_user)
-    if user.role == UserRole.LIMITED:
-        logger.warning(f"WS auth: user {user.email} has LIMITED role")
-        raise BasicAuthenticationError(
-            detail="Access denied. User role is LIMITED. BASIC or higher permissions are required.",
-        )
-
-    logger.debug(f"WS auth: authenticated {user.email}")
-    return user
-
-
 def get_default_admin_user_emails_() -> list[str]:
     # No default seeding available for Onyx MIT
     return []
@@ -1727,7 +1608,6 @@ STATE_TOKEN_AUDIENCE = "fastapi-users:oauth-state"
 STATE_TOKEN_LIFETIME_SECONDS = 3600
 CSRF_TOKEN_KEY = "csrftoken"
 CSRF_TOKEN_COOKIE_NAME = "fastapiusersoauthcsrf"
-PKCE_COOKIE_NAME_PREFIX = "fastapiusersoauthpkce"
 
 
 class OAuth2AuthorizeResponse(BaseModel):
@@ -1748,21 +1628,6 @@ def generate_csrf_token() -> str:
     return secrets.token_urlsafe(32)
 
 
-def _base64url_encode(data: bytes) -> str:
-    return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
-
-
-def generate_pkce_pair() -> tuple[str, str]:
-    verifier = secrets.token_urlsafe(64)
-    challenge = _base64url_encode(hashlib.sha256(verifier.encode("ascii")).digest())
-    return verifier, challenge
-
-
-def get_pkce_cookie_name(state: str) -> str:
-    state_hash = hashlib.sha256(state.encode("utf-8")).hexdigest()
-    return f"{PKCE_COOKIE_NAME_PREFIX}_{state_hash}"
-
-
 # refer to https://github.com/fastapi-users/fastapi-users/blob/42ddc241b965475390e2bce887b084152ae1a2cd/fastapi_users/fastapi_users.py#L91
 def create_onyx_oauth_router(
     oauth_client: BaseOAuth2,
@@ -1771,7 +1636,6 @@ def create_onyx_oauth_router(
     redirect_url: Optional[str] = None,
     associate_by_email: bool = False,
     is_verified_by_default: bool = False,
-    enable_pkce: bool = False,
 ) -> APIRouter:
     return get_oauth_router(
         oauth_client,
@@ -1781,7 +1645,6 @@ def create_onyx_oauth_router(
         redirect_url,
         associate_by_email,
         is_verified_by_default,
-        enable_pkce=enable_pkce,
     )
 
 
@@ -1800,7 +1663,6 @@ def get_oauth_router(
     csrf_token_cookie_secure: Optional[bool] = None,
     csrf_token_cookie_httponly: bool = True,
     csrf_token_cookie_samesite: Optional[Literal["lax", "strict", "none"]] = "lax",
-    enable_pkce: bool = False,
 ) -> APIRouter:
     """Generate a router with the OAuth routes."""
     router = APIRouter()
@@ -1817,13 +1679,6 @@ def get_oauth_router(
             route_name=callback_route_name,
         )
 
-    async def null_access_token_state() -> tuple[OAuth2Token, Optional[str]] | None:
-        return None
-
-    access_token_state_dependency = (
-        oauth2_authorize_callback if not enable_pkce else null_access_token_state
-    )
-
     if csrf_token_cookie_secure is None:
         csrf_token_cookie_secure = WEB_DOMAIN.startswith("https")
 
@@ -1857,26 +1712,13 @@ def get_oauth_router(
             CSRF_TOKEN_KEY: csrf_token,
         }
         state = generate_state_token(state_data, state_secret)
-        pkce_cookie: tuple[str, str] | None = None
 
-        if enable_pkce:
-            code_verifier, code_challenge = generate_pkce_pair()
-            pkce_cookie_name = get_pkce_cookie_name(state)
-            pkce_cookie = (pkce_cookie_name, code_verifier)
-            authorization_url = await oauth_client.get_authorization_url(
-                authorize_redirect_url,
-                state,
-                scopes,
-                code_challenge=code_challenge,
-                code_challenge_method="S256",
-            )
-        else:
-            # Get the basic authorization URL
-            authorization_url = await oauth_client.get_authorization_url(
-                authorize_redirect_url,
-                state,
-                scopes,
-            )
+        # Get the basic authorization URL
+        authorization_url = await oauth_client.get_authorization_url(
+            authorize_redirect_url,
+            state,
+            scopes,
+        )
 
         # For Google OAuth, add parameters to request refresh tokens
         if oauth_client.name == "google":
@@ -1884,15 +1726,11 @@ def get_oauth_router(
                 authorization_url, {"access_type": "offline", "prompt": "consent"}
             )
 
-        def set_oauth_cookie(
-            target_response: Response,
-            *,
-            key: str,
-            value: str,
-        ) -> None:
-            target_response.set_cookie(
-                key=key,
-                value=value,
+        if redirect:
+            redirect_response = RedirectResponse(authorization_url, status_code=302)
+            redirect_response.set_cookie(
+                key=csrf_token_cookie_name,
+                value=csrf_token,
                 max_age=STATE_TOKEN_LIFETIME_SECONDS,
                 path=csrf_token_cookie_path,
                 domain=csrf_token_cookie_domain,
@@ -1900,28 +1738,18 @@ def get_oauth_router(
                 httponly=csrf_token_cookie_httponly,
                 samesite=csrf_token_cookie_samesite,
             )
+            return redirect_response
 
-        response_with_cookies: Response
-        if redirect:
-            response_with_cookies = RedirectResponse(authorization_url, status_code=302)
-        else:
-            response_with_cookies = response
-
-        set_oauth_cookie(
-            response_with_cookies,
+        response.set_cookie(
             key=csrf_token_cookie_name,
             value=csrf_token,
+            max_age=STATE_TOKEN_LIFETIME_SECONDS,
+            path=csrf_token_cookie_path,
+            domain=csrf_token_cookie_domain,
+            secure=csrf_token_cookie_secure,
+            httponly=csrf_token_cookie_httponly,
+            samesite=csrf_token_cookie_samesite,
         )
-        if pkce_cookie is not None:
-            pkce_cookie_name, code_verifier = pkce_cookie
-            set_oauth_cookie(
-                response_with_cookies,
-                key=pkce_cookie_name,
-                value=code_verifier,
-            )
-
-        if redirect:
-            return response_with_cookies
 
         return OAuth2AuthorizeResponse(authorization_url=authorization_url)
 
@@ -1952,242 +1780,119 @@ def get_oauth_router(
     )
     async def callback(
         request: Request,
-        access_token_state: Tuple[OAuth2Token, Optional[str]] | None = Depends(
-            access_token_state_dependency
+        access_token_state: Tuple[OAuth2Token, str] = Depends(
+            oauth2_authorize_callback
         ),
-        code: Optional[str] = None,
-        state: Optional[str] = None,
-        error: Optional[str] = None,
         user_manager: BaseUserManager[models.UP, models.ID] = Depends(get_user_manager),
         strategy: Strategy[models.UP, models.ID] = Depends(backend.get_strategy),
-    ) -> Response:
-        pkce_cookie_name: str | None = None
+    ) -> RedirectResponse:
+        token, state = access_token_state
+        account_id, account_email = await oauth_client.get_id_email(
+            token["access_token"]
+        )
 
-        def delete_pkce_cookie(response: Response) -> None:
-            if enable_pkce and pkce_cookie_name:
-                response.delete_cookie(
-                    key=pkce_cookie_name,
-                    path=csrf_token_cookie_path,
-                    domain=csrf_token_cookie_domain,
-                    secure=csrf_token_cookie_secure,
-                    httponly=csrf_token_cookie_httponly,
-                    samesite=csrf_token_cookie_samesite,
-                )
-
-        def build_error_response(exc: OnyxError) -> JSONResponse:
-            log_onyx_error(exc)
-            error_response = onyx_error_to_json_response(exc)
-            delete_pkce_cookie(error_response)
-            return error_response
-
-        def decode_and_validate_state(state_value: str) -> Dict[str, str]:
-            try:
-                state_data = decode_jwt(
-                    state_value, state_secret, [STATE_TOKEN_AUDIENCE]
-                )
-            except jwt.DecodeError:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    getattr(
-                        ErrorCode,
-                        "ACCESS_TOKEN_DECODE_ERROR",
-                        "ACCESS_TOKEN_DECODE_ERROR",
-                    ),
-                )
-            except jwt.ExpiredSignatureError:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    getattr(
-                        ErrorCode,
-                        "ACCESS_TOKEN_ALREADY_EXPIRED",
-                        "ACCESS_TOKEN_ALREADY_EXPIRED",
-                    ),
-                )
-            except jwt.PyJWTError:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    getattr(
-                        ErrorCode,
-                        "ACCESS_TOKEN_DECODE_ERROR",
-                        "ACCESS_TOKEN_DECODE_ERROR",
-                    ),
-                )
-
-            cookie_csrf_token = request.cookies.get(csrf_token_cookie_name)
-            state_csrf_token = state_data.get(CSRF_TOKEN_KEY)
-            if (
-                not cookie_csrf_token
-                or not state_csrf_token
-                or not secrets.compare_digest(cookie_csrf_token, state_csrf_token)
-            ):
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    getattr(ErrorCode, "OAUTH_INVALID_STATE", "OAUTH_INVALID_STATE"),
-                )
-
-            return state_data
-
-        token: OAuth2Token
-        state_data: Dict[str, str]
-
-        # `code`, `state`, and `error` are read directly only in the PKCE path.
-        # In the non-PKCE path, `oauth2_authorize_callback` consumes them.
-        if enable_pkce:
-            if state is not None:
-                pkce_cookie_name = get_pkce_cookie_name(state)
-
-            if error is not None:
-                return build_error_response(
-                    OnyxError(
-                        OnyxErrorCode.VALIDATION_ERROR,
-                        "Authorization request failed or was denied",
-                    )
-                )
-            if code is None:
-                return build_error_response(
-                    OnyxError(
-                        OnyxErrorCode.VALIDATION_ERROR,
-                        "Missing authorization code in OAuth callback",
-                    )
-                )
-            if state is None:
-                return build_error_response(
-                    OnyxError(
-                        OnyxErrorCode.VALIDATION_ERROR,
-                        "Missing state parameter in OAuth callback",
-                    )
-                )
-
-            state_value = state
-
-            if redirect_url is not None:
-                callback_redirect_url = redirect_url
-            else:
-                callback_path = request.app.url_path_for(callback_route_name)
-                callback_redirect_url = f"{WEB_DOMAIN}{callback_path}"
-
-            code_verifier = request.cookies.get(cast(str, pkce_cookie_name))
-            if not code_verifier:
-                return build_error_response(
-                    OnyxError(
-                        OnyxErrorCode.VALIDATION_ERROR,
-                        "Missing PKCE verifier cookie in OAuth callback",
-                    )
-                )
-
-            try:
-                state_data = decode_and_validate_state(state_value)
-            except OnyxError as e:
-                return build_error_response(e)
-
-            try:
-                token = await oauth_client.get_access_token(
-                    code, callback_redirect_url, code_verifier
-                )
-            except GetAccessTokenError:
-                return build_error_response(
-                    OnyxError(
-                        OnyxErrorCode.VALIDATION_ERROR,
-                        "Authorization code exchange failed",
-                    )
-                )
-        else:
-            if access_token_state is None:
-                raise OnyxError(
-                    OnyxErrorCode.INTERNAL_ERROR, "Missing OAuth callback state"
-                )
-            token, callback_state = access_token_state
-            if callback_state is None:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    "Missing state parameter in OAuth callback",
-                )
-            state_data = decode_and_validate_state(callback_state)
-
-        async def complete_login_flow(
-            token: OAuth2Token, state_data: Dict[str, str]
-        ) -> RedirectResponse:
-            account_id, account_email = await oauth_client.get_id_email(
-                token["access_token"]
+        if account_email is None:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=ErrorCode.OAUTH_NOT_AVAILABLE_EMAIL,
             )
 
-            if account_email is None:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    ErrorCode.OAUTH_NOT_AVAILABLE_EMAIL,
-                )
+        try:
+            state_data = decode_jwt(state, state_secret, [STATE_TOKEN_AUDIENCE])
+        except jwt.DecodeError:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=getattr(
+                    ErrorCode, "ACCESS_TOKEN_DECODE_ERROR", "ACCESS_TOKEN_DECODE_ERROR"
+                ),
+            )
+        except jwt.ExpiredSignatureError:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=getattr(
+                    ErrorCode,
+                    "ACCESS_TOKEN_ALREADY_EXPIRED",
+                    "ACCESS_TOKEN_ALREADY_EXPIRED",
+                ),
+            )
 
-            next_url = state_data.get("next_url", "/")
-            referral_source = state_data.get("referral_source", None)
-            try:
-                tenant_id = fetch_ee_implementation_or_noop(
-                    "onyx.server.tenants.user_mapping", "get_tenant_id_for_email", None
-                )(account_email)
-            except exceptions.UserNotExists:
-                tenant_id = None
+        cookie_csrf_token = request.cookies.get(csrf_token_cookie_name)
+        state_csrf_token = state_data.get(CSRF_TOKEN_KEY)
+        if (
+            not cookie_csrf_token
+            or not state_csrf_token
+            or not secrets.compare_digest(cookie_csrf_token, state_csrf_token)
+        ):
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=getattr(ErrorCode, "OAUTH_INVALID_STATE", "OAUTH_INVALID_STATE"),
+            )
 
-            request.state.referral_source = referral_source
+        next_url = state_data.get("next_url", "/")
+        referral_source = state_data.get("referral_source", None)
+        try:
+            tenant_id = fetch_ee_implementation_or_noop(
+                "onyx.server.tenants.user_mapping", "get_tenant_id_for_email", None
+            )(account_email)
+        except exceptions.UserNotExists:
+            tenant_id = None
 
-            # Proceed to authenticate or create the user
-            try:
-                user = await user_manager.oauth_callback(
-                    oauth_client.name,
-                    token["access_token"],
-                    account_id,
-                    account_email,
-                    token.get("expires_at"),
-                    token.get("refresh_token"),
-                    request,
-                    associate_by_email=associate_by_email,
-                    is_verified_by_default=is_verified_by_default,
-                )
-            except UserAlreadyExists:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    ErrorCode.OAUTH_USER_ALREADY_EXISTS,
-                )
+        request.state.referral_source = referral_source
 
-            if not user.is_active:
-                raise OnyxError(
-                    OnyxErrorCode.VALIDATION_ERROR,
-                    ErrorCode.LOGIN_BAD_CREDENTIALS,
-                )
+        # Proceed to authenticate or create the user
+        try:
+            user = await user_manager.oauth_callback(
+                oauth_client.name,
+                token["access_token"],
+                account_id,
+                account_email,
+                token.get("expires_at"),
+                token.get("refresh_token"),
+                request,
+                associate_by_email=associate_by_email,
+                is_verified_by_default=is_verified_by_default,
+            )
+        except UserAlreadyExists:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=ErrorCode.OAUTH_USER_ALREADY_EXISTS,
+            )
 
-            # Login user
-            response = await backend.login(strategy, user)
-            await user_manager.on_after_login(user, request, response)
+        if not user.is_active:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=ErrorCode.LOGIN_BAD_CREDENTIALS,
+            )
 
-            # Prepare redirect response
-            if tenant_id is None:
-                # Use URL utility to add parameters
-                redirect_destination = add_url_params(next_url, {"new_team": "true"})
-                redirect_response = RedirectResponse(
-                    redirect_destination, status_code=302
-                )
+        # Login user
+        response = await backend.login(strategy, user)
+        await user_manager.on_after_login(user, request, response)
+
+        # Prepare redirect response
+        if tenant_id is None:
+            # Use URL utility to add parameters
+            redirect_url = add_url_params(next_url, {"new_team": "true"})
+            redirect_response = RedirectResponse(redirect_url, status_code=302)
+        else:
+            # No parameters to add
+            redirect_response = RedirectResponse(next_url, status_code=302)
+
+        # Copy headers from auth response to redirect response, with special handling for Set-Cookie
+        for header_name, header_value in response.headers.items():
+            # FastAPI can have multiple Set-Cookie headers as a list
+            if header_name.lower() == "set-cookie" and isinstance(header_value, list):
+                for cookie_value in header_value:
+                    redirect_response.headers.append(header_name, cookie_value)
             else:
-                # No parameters to add
-                redirect_response = RedirectResponse(next_url, status_code=302)
-
-            # Copy headers from auth response to redirect response, with special handling for Set-Cookie
-            for header_name, header_value in response.headers.items():
-                header_name_lower = header_name.lower()
-                if header_name_lower == "set-cookie":
-                    redirect_response.headers.append(header_name, header_value)
-                    continue
-                if header_name_lower in {"location", "content-length"}:
-                    continue
                 redirect_response.headers[header_name] = header_value
 
-            return redirect_response
+        if hasattr(response, "body"):
+            redirect_response.body = response.body
+        if hasattr(response, "status_code"):
+            redirect_response.status_code = response.status_code
+        if hasattr(response, "media_type"):
+            redirect_response.media_type = response.media_type
 
-        if enable_pkce:
-            try:
-                redirect_response = await complete_login_flow(token, state_data)
-            except OnyxError as e:
-                return build_error_response(e)
-            delete_pkce_cookie(redirect_response)
-            return redirect_response
-
-        return await complete_login_flow(token, state_data)
+        return redirect_response
 
     return router

backend/onyx/background/celery/apps/background.py

@@ -0,0 +1,142 @@
+from typing import Any
+from typing import cast
+
+from celery import Celery
+from celery import signals
+from celery import Task
+from celery.apps.worker import Worker
+from celery.signals import celeryd_init
+from celery.signals import worker_init
+from celery.signals import worker_process_init
+from celery.signals import worker_ready
+from celery.signals import worker_shutdown
+
+import onyx.background.celery.apps.app_base as app_base
+from onyx.background.celery.celery_utils import httpx_init_vespa_pool
+from onyx.configs.app_configs import MANAGED_VESPA
+from onyx.configs.app_configs import VESPA_CLOUD_CERT_PATH
+from onyx.configs.app_configs import VESPA_CLOUD_KEY_PATH
+from onyx.configs.constants import POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME
+from onyx.db.engine.sql_engine import SqlEngine
+from onyx.utils.logger import setup_logger
+from shared_configs.configs import MULTI_TENANT
+
+
+logger = setup_logger()
+
+celery_app = Celery(__name__)
+celery_app.config_from_object("onyx.background.celery.configs.background")
+celery_app.Task = app_base.TenantAwareTask  # type: ignore [misc]
+
+
+@signals.task_prerun.connect
+def on_task_prerun(
+    sender: Any | None = None,
+    task_id: str | None = None,
+    task: Task | None = None,
+    args: tuple | None = None,
+    kwargs: dict | None = None,
+    **kwds: Any,
+) -> None:
+    app_base.on_task_prerun(sender, task_id, task, args, kwargs, **kwds)
+
+
+@signals.task_postrun.connect
+def on_task_postrun(
+    sender: Any | None = None,
+    task_id: str | None = None,
+    task: Task | None = None,
+    args: tuple | None = None,
+    kwargs: dict | None = None,
+    retval: Any | None = None,
+    state: str | None = None,
+    **kwds: Any,
+) -> None:
+    app_base.on_task_postrun(sender, task_id, task, args, kwargs, retval, state, **kwds)
+
+
+@celeryd_init.connect
+def on_celeryd_init(sender: str, conf: Any = None, **kwargs: Any) -> None:
+    app_base.on_celeryd_init(sender, conf, **kwargs)
+
+
+@worker_init.connect
+def on_worker_init(sender: Worker, **kwargs: Any) -> None:
+    EXTRA_CONCURRENCY = 8  # small extra fudge factor for connection limits
+
+    logger.info("worker_init signal received for consolidated background worker.")
+
+    SqlEngine.set_app_name(POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME)
+    pool_size = cast(int, sender.concurrency)  # type: ignore
+    SqlEngine.init_engine(pool_size=pool_size, max_overflow=EXTRA_CONCURRENCY)
+
+    # Initialize Vespa httpx pool (needed for light worker tasks)
+    if MANAGED_VESPA:
+        httpx_init_vespa_pool(
+            sender.concurrency + EXTRA_CONCURRENCY,  # type: ignore
+            ssl_cert=VESPA_CLOUD_CERT_PATH,
+            ssl_key=VESPA_CLOUD_KEY_PATH,
+        )
+    else:
+        httpx_init_vespa_pool(sender.concurrency + EXTRA_CONCURRENCY)  # type: ignore
+
+    app_base.wait_for_redis(sender, **kwargs)
+    app_base.wait_for_db(sender, **kwargs)
+    app_base.wait_for_vespa_or_shutdown(sender, **kwargs)
+
+    # Less startup checks in multi-tenant case
+    if MULTI_TENANT:
+        return
+
+    app_base.on_secondary_worker_init(sender, **kwargs)
+
+
+@worker_ready.connect
+def on_worker_ready(sender: Any, **kwargs: Any) -> None:
+    app_base.on_worker_ready(sender, **kwargs)
+
+
+@worker_shutdown.connect
+def on_worker_shutdown(sender: Any, **kwargs: Any) -> None:
+    app_base.on_worker_shutdown(sender, **kwargs)
+
+
+@worker_process_init.connect
+def init_worker(**kwargs: Any) -> None:  # noqa: ARG001
+    SqlEngine.reset_engine()
+
+
+@signals.setup_logging.connect
+def on_setup_logging(
+    loglevel: Any, logfile: Any, format: Any, colorize: Any, **kwargs: Any
+) -> None:
+    app_base.on_setup_logging(loglevel, logfile, format, colorize, **kwargs)
+
+
+base_bootsteps = app_base.get_bootsteps()
+for bootstep in base_bootsteps:
+    celery_app.steps["worker"].add(bootstep)
+
+celery_app.autodiscover_tasks(
+    app_base.filter_task_modules(
+        [
+            # Original background worker tasks
+            "onyx.background.celery.tasks.pruning",
+            "onyx.background.celery.tasks.monitoring",
+            "onyx.background.celery.tasks.user_file_processing",
+            "onyx.background.celery.tasks.llm_model_update",
+            # Light worker tasks
+            "onyx.background.celery.tasks.shared",
+            "onyx.background.celery.tasks.vespa",
+            "onyx.background.celery.tasks.connector_deletion",
+            "onyx.background.celery.tasks.doc_permission_syncing",
+            "onyx.background.celery.tasks.opensearch_migration",
+            # Docprocessing worker tasks
+            "onyx.background.celery.tasks.docprocessing",
+            # Docfetching worker tasks
+            "onyx.background.celery.tasks.docfetching",
+            # Sandbox cleanup tasks (isolated in build feature)
+            "onyx.server.features.build.sandbox.tasks",
+        ]
+    )
+)

backend/onyx/background/celery/celery_utils.py

@@ -39,13 +39,9 @@ CT = TypeVar("CT", bound=ConnectorCheckpoint)
 
 
 class SlimConnectorExtractionResult(BaseModel):
-    """Result of extracting document IDs and hierarchy nodes from a connector.
+    """Result of extracting document IDs and hierarchy nodes from a connector."""
 
-    raw_id_to_parent maps document ID → parent_hierarchy_raw_node_id (or None).
-    Use raw_id_to_parent.keys() wherever the old set of IDs was needed.
-    """
-
-    raw_id_to_parent: dict[str, str | None]
+    doc_ids: set[str]
     hierarchy_nodes: list[HierarchyNode]
 
 
@@ -97,34 +93,30 @@ def _get_failure_id(failure: ConnectorFailure) -> str | None:
     return None
 
 
-class BatchResult(BaseModel):
-    raw_id_to_parent: dict[str, str | None]
-    hierarchy_nodes: list[HierarchyNode]
-
-
 def _extract_from_batch(
     doc_list: Sequence[Document | SlimDocument | HierarchyNode | ConnectorFailure],
-) -> BatchResult:
-    """Separate a batch into document IDs (with parent mapping) and hierarchy nodes.
+) -> tuple[set[str], list[HierarchyNode]]:
+    """Separate a batch into document IDs and hierarchy nodes.
 
     ConnectorFailure items have their failed document/entity IDs added to the
-    ID dict so that failed-to-retrieve documents are not accidentally pruned.
+    ID set so that failed-to-retrieve documents are not accidentally pruned.
     """
-    ids: dict[str, str | None] = {}
+    ids: set[str] = set()
     hierarchy_nodes: list[HierarchyNode] = []
     for item in doc_list:
         if isinstance(item, HierarchyNode):
             hierarchy_nodes.append(item)
+            ids.add(item.raw_node_id)
         elif isinstance(item, ConnectorFailure):
             failed_id = _get_failure_id(item)
             if failed_id:
-                ids[failed_id] = None
+                ids.add(failed_id)
             logger.warning(
                 f"Failed to retrieve document {failed_id}: " f"{item.failure_message}"
             )
         else:
-            ids[item.id] = item.parent_hierarchy_raw_node_id
-    return BatchResult(raw_id_to_parent=ids, hierarchy_nodes=hierarchy_nodes)
+            ids.add(item.id)
+    return ids, hierarchy_nodes
 
 
 def extract_ids_from_runnable_connector(
@@ -140,7 +132,7 @@ def extract_ids_from_runnable_connector(
 
     Optionally, a callback can be passed to handle the length of each document batch.
     """
-    all_raw_id_to_parent: dict[str, str | None] = {}
+    all_connector_doc_ids: set[str] = set()
     all_hierarchy_nodes: list[HierarchyNode] = []
 
     # Sequence (covariant) lets all the specific list[...] iterator types unify here
@@ -185,18 +177,15 @@ def extract_ids_from_runnable_connector(
                 "extract_ids_from_runnable_connector: Stop signal detected"
             )
 
-        batch_result = _extract_from_batch(doc_list)
-        batch_ids = batch_result.raw_id_to_parent
-        batch_nodes = batch_result.hierarchy_nodes
-        doc_batch_processing_func(batch_ids)
-        all_raw_id_to_parent.update(batch_ids)
+        batch_ids, batch_nodes = _extract_from_batch(doc_list)
+        all_connector_doc_ids.update(doc_batch_processing_func(batch_ids))
         all_hierarchy_nodes.extend(batch_nodes)
 
         if callback:
             callback.progress("extract_ids_from_runnable_connector", len(batch_ids))
 
     return SlimConnectorExtractionResult(
-        raw_id_to_parent=all_raw_id_to_parent,
+        doc_ids=all_connector_doc_ids,
         hierarchy_nodes=all_hierarchy_nodes,
     )
 

backend/onyx/background/celery/configs/background.py

@@ -0,0 +1,23 @@
+import onyx.background.celery.configs.base as shared_config
+from onyx.configs.app_configs import CELERY_WORKER_BACKGROUND_CONCURRENCY
+
+broker_url = shared_config.broker_url
+broker_connection_retry_on_startup = shared_config.broker_connection_retry_on_startup
+broker_pool_limit = shared_config.broker_pool_limit
+broker_transport_options = shared_config.broker_transport_options
+
+redis_socket_keepalive = shared_config.redis_socket_keepalive
+redis_retry_on_timeout = shared_config.redis_retry_on_timeout
+redis_backend_health_check_interval = shared_config.redis_backend_health_check_interval
+
+result_backend = shared_config.result_backend
+result_expires = shared_config.result_expires  # 86400 seconds is the default
+
+task_default_priority = shared_config.task_default_priority
+task_acks_late = shared_config.task_acks_late
+
+worker_concurrency = CELERY_WORKER_BACKGROUND_CONCURRENCY
+worker_pool = "threads"
+# Increased from 1 to 4 to handle fast light worker tasks more efficiently
+# This allows the worker to prefetch multiple tasks per thread
+worker_prefetch_multiplier = 4

backend/onyx/background/celery/tasks/hierarchyfetching/tasks.py

@@ -40,7 +40,6 @@ from onyx.db.connector_credential_pair import get_connector_credential_pair_from
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
 from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
-from onyx.db.hierarchy import upsert_hierarchy_node_cc_pair_entries
 from onyx.db.hierarchy import upsert_hierarchy_nodes_batch
 from onyx.db.models import ConnectorCredentialPair
 from onyx.redis.redis_hierarchy import cache_hierarchy_nodes_batch
@@ -290,14 +289,6 @@ def _run_hierarchy_extraction(
             is_connector_public=is_connector_public,
         )
 
-        upsert_hierarchy_node_cc_pair_entries(
-            db_session=db_session,
-            hierarchy_node_ids=[n.id for n in upserted_nodes],
-            connector_id=cc_pair.connector_id,
-            credential_id=cc_pair.credential_id,
-            commit=True,
-        )
-
         # Cache in Redis for fast ancestor resolution
         cache_entries = [
             HierarchyNodeCacheEntry.from_db_model(node) for node in upserted_nodes

backend/onyx/background/celery/tasks/opensearch_migration/constants.py

@@ -11,9 +11,6 @@
 # lock after its cleanup which happens at most after its soft timeout.
 
 # Constants corresponding to migrate_documents_from_vespa_to_opensearch_task.
-from onyx.configs.app_configs import OPENSEARCH_MIGRATION_GET_VESPA_CHUNKS_PAGE_SIZE
-
-
 MIGRATION_TASK_SOFT_TIME_LIMIT_S = 60 * 5  # 5 minutes.
 MIGRATION_TASK_TIME_LIMIT_S = 60 * 6  # 6 minutes.
 # The maximum time the lock can be held for. Will automatically be released
@@ -47,7 +44,7 @@ TOTAL_ALLOWABLE_DOC_MIGRATION_ATTEMPTS_BEFORE_PERMANENT_FAILURE = 15
 
 # WARNING: Do not change these values without knowing what changes also need to
 # be made to OpenSearchTenantMigrationRecord.
-GET_VESPA_CHUNKS_PAGE_SIZE = OPENSEARCH_MIGRATION_GET_VESPA_CHUNKS_PAGE_SIZE
+GET_VESPA_CHUNKS_PAGE_SIZE = 500
 GET_VESPA_CHUNKS_SLICE_COUNT = 4
 
 # String used to indicate in the vespa_visit_continuation_token mapping that the

backend/onyx/background/celery/tasks/opensearch_migration/tasks.py

@@ -30,7 +30,6 @@ from onyx.background.celery.tasks.opensearch_migration.transformer import (
     transform_vespa_chunks_to_opensearch_chunks,
 )
 from onyx.configs.app_configs import ENABLE_OPENSEARCH_INDEXING_FOR_ONYX
-from onyx.configs.app_configs import VESPA_MIGRATION_REQUEST_TIMEOUT_S
 from onyx.configs.constants import OnyxCeleryTask
 from onyx.configs.constants import OnyxRedisLocks
 from onyx.db.engine.sql_engine import get_session_with_current_tenant
@@ -48,7 +47,6 @@ from onyx.document_index.interfaces_new import TenantState
 from onyx.document_index.opensearch.opensearch_document_index import (
     OpenSearchDocumentIndex,
 )
-from onyx.document_index.vespa.shared_utils.utils import get_vespa_http_client
 from onyx.document_index.vespa.vespa_document_index import VespaDocumentIndex
 from onyx.indexing.models import IndexingSetting
 from onyx.redis.redis_pool import get_redis_client
@@ -148,12 +146,7 @@ def migrate_chunks_from_vespa_to_opensearch_task(
             task_logger.error(err_str)
             return False
 
-        with (
-            get_session_with_current_tenant() as db_session,
-            get_vespa_http_client(
-                timeout=VESPA_MIGRATION_REQUEST_TIMEOUT_S
-            ) as vespa_client,
-        ):
+        with get_session_with_current_tenant() as db_session:
             try_insert_opensearch_tenant_migration_record_with_commit(db_session)
             search_settings = get_current_search_settings(db_session)
             tenant_state = TenantState(tenant_id=tenant_id, multitenant=MULTI_TENANT)
@@ -168,7 +161,6 @@ def migrate_chunks_from_vespa_to_opensearch_task(
                 index_name=search_settings.index_name,
                 tenant_state=tenant_state,
                 large_chunks_enabled=False,
-                httpx_client=vespa_client,
             )
 
             sanitized_doc_start_time = time.monotonic()

backend/onyx/background/celery/tasks/pruning/tasks.py

@@ -29,7 +29,6 @@ from onyx.configs.constants import CELERY_GENERIC_BEAT_LOCK_TIMEOUT
 from onyx.configs.constants import CELERY_PRUNING_LOCK_TIMEOUT
 from onyx.configs.constants import CELERY_TASK_WAIT_FOR_FENCE_TIMEOUT
 from onyx.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
-from onyx.configs.constants import DocumentSource
 from onyx.configs.constants import OnyxCeleryPriority
 from onyx.configs.constants import OnyxCeleryQueues
 from onyx.configs.constants import OnyxCeleryTask
@@ -48,15 +47,8 @@ from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
 from onyx.db.enums import SyncStatus
 from onyx.db.enums import SyncType
-from onyx.db.hierarchy import delete_orphaned_hierarchy_nodes
-from onyx.db.hierarchy import link_hierarchy_nodes_to_documents
-from onyx.db.hierarchy import remove_stale_hierarchy_node_cc_pair_entries
-from onyx.db.hierarchy import reparent_orphaned_hierarchy_nodes
-from onyx.db.hierarchy import update_document_parent_hierarchy_nodes
-from onyx.db.hierarchy import upsert_hierarchy_node_cc_pair_entries
 from onyx.db.hierarchy import upsert_hierarchy_nodes_batch
 from onyx.db.models import ConnectorCredentialPair
-from onyx.db.models import HierarchyNode as DBHierarchyNode
 from onyx.db.sync_record import insert_sync_record
 from onyx.db.sync_record import update_sync_record_status
 from onyx.db.tag import delete_orphan_tags__no_commit
@@ -65,9 +57,6 @@ from onyx.redis.redis_connector_prune import RedisConnectorPrune
 from onyx.redis.redis_connector_prune import RedisConnectorPrunePayload
 from onyx.redis.redis_hierarchy import cache_hierarchy_nodes_batch
 from onyx.redis.redis_hierarchy import ensure_source_node_exists
-from onyx.redis.redis_hierarchy import evict_hierarchy_nodes_from_cache
-from onyx.redis.redis_hierarchy import get_node_id_from_raw_id
-from onyx.redis.redis_hierarchy import get_source_node_id_from_cache
 from onyx.redis.redis_hierarchy import HierarchyNodeCacheEntry
 from onyx.redis.redis_pool import get_redis_client
 from onyx.redis.redis_pool import get_redis_replica_client
@@ -124,38 +113,6 @@ class PruneCallback(IndexingCallbackBase):
         super().progress(tag, amount)
 
 
-def _resolve_and_update_document_parents(
-    db_session: Session,
-    redis_client: Redis,
-    source: DocumentSource,
-    raw_id_to_parent: dict[str, str | None],
-) -> None:
-    """Resolve parent_hierarchy_raw_node_id → parent_hierarchy_node_id for
-    each document and bulk-update the DB. Mirrors the resolution logic in
-    run_docfetching.py."""
-    source_node_id = get_source_node_id_from_cache(redis_client, db_session, source)
-
-    resolved: dict[str, int | None] = {}
-    for doc_id, raw_parent_id in raw_id_to_parent.items():
-        if raw_parent_id is None:
-            continue
-        node_id, found = get_node_id_from_raw_id(redis_client, source, raw_parent_id)
-        resolved[doc_id] = node_id if found else source_node_id
-
-    if not resolved:
-        return
-
-    update_document_parent_hierarchy_nodes(
-        db_session=db_session,
-        doc_parent_map=resolved,
-        commit=True,
-    )
-    task_logger.info(
-        f"Pruning: resolved and updated parent hierarchy for "
-        f"{len(resolved)} documents (source={source.value})"
-    )
-
-
 """Jobs / utils for kicking off pruning tasks."""
 
 
@@ -578,42 +535,33 @@ def connector_pruning_generator_task(
             extraction_result = extract_ids_from_runnable_connector(
                 runnable_connector, callback
             )
-            all_connector_doc_ids = extraction_result.raw_id_to_parent
+            all_connector_doc_ids = extraction_result.doc_ids
 
             # Process hierarchy nodes (same as docfetching):
             # upsert to Postgres and cache in Redis
-            source = cc_pair.connector.source
-            redis_client = get_redis_client(tenant_id=tenant_id)
-
-            ensure_source_node_exists(redis_client, db_session, source)
-
-            upserted_nodes: list[DBHierarchyNode] = []
             if extraction_result.hierarchy_nodes:
                 is_connector_public = cc_pair.access_type == AccessType.PUBLIC
 
+                redis_client = get_redis_client(tenant_id=tenant_id)
+                ensure_source_node_exists(
+                    redis_client, db_session, cc_pair.connector.source
+                )
+
                 upserted_nodes = upsert_hierarchy_nodes_batch(
                     db_session=db_session,
                     nodes=extraction_result.hierarchy_nodes,
-                    source=source,
+                    source=cc_pair.connector.source,
                     commit=True,
                     is_connector_public=is_connector_public,
                 )
 
-                upsert_hierarchy_node_cc_pair_entries(
-                    db_session=db_session,
-                    hierarchy_node_ids=[n.id for n in upserted_nodes],
-                    connector_id=connector_id,
-                    credential_id=credential_id,
-                    commit=True,
-                )
-
                 cache_entries = [
                     HierarchyNodeCacheEntry.from_db_model(node)
                     for node in upserted_nodes
                 ]
                 cache_hierarchy_nodes_batch(
                     redis_client=redis_client,
-                    source=source,
+                    source=cc_pair.connector.source,
                     entries=cache_entries,
                 )
 
@@ -622,25 +570,6 @@ def connector_pruning_generator_task(
                     f"hierarchy nodes for cc_pair={cc_pair_id}"
                 )
 
-            # Resolve parent_hierarchy_raw_node_id → parent_hierarchy_node_id
-            # and bulk-update documents, mirroring the docfetching resolution
-            _resolve_and_update_document_parents(
-                db_session=db_session,
-                redis_client=redis_client,
-                source=source,
-                raw_id_to_parent=all_connector_doc_ids,
-            )
-
-            # Link hierarchy nodes to documents for sources where pages can be
-            # both hierarchy nodes AND documents (e.g. Notion, Confluence)
-            all_doc_id_list = list(all_connector_doc_ids.keys())
-            link_hierarchy_nodes_to_documents(
-                db_session=db_session,
-                document_ids=all_doc_id_list,
-                source=source,
-                commit=True,
-            )
-
             # a list of docs in our local index
             all_indexed_document_ids = {
                 doc.id
@@ -652,9 +581,7 @@ def connector_pruning_generator_task(
             }
 
             # generate list of docs to remove (no longer in the source)
-            doc_ids_to_remove = list(
-                all_indexed_document_ids - all_connector_doc_ids.keys()
-            )
+            doc_ids_to_remove = list(all_indexed_document_ids - all_connector_doc_ids)
 
             task_logger.info(
                 "Pruning set collected: "
@@ -678,43 +605,6 @@ def connector_pruning_generator_task(
             )
 
             redis_connector.prune.generator_complete = tasks_generated
-
-            # --- Hierarchy node pruning ---
-            live_node_ids = {n.id for n in upserted_nodes}
-            stale_removed = remove_stale_hierarchy_node_cc_pair_entries(
-                db_session=db_session,
-                connector_id=connector_id,
-                credential_id=credential_id,
-                live_hierarchy_node_ids=live_node_ids,
-                commit=True,
-            )
-            deleted_raw_ids = delete_orphaned_hierarchy_nodes(
-                db_session=db_session,
-                source=source,
-                commit=True,
-            )
-            reparented_nodes = reparent_orphaned_hierarchy_nodes(
-                db_session=db_session,
-                source=source,
-                commit=True,
-            )
-            if deleted_raw_ids:
-                evict_hierarchy_nodes_from_cache(redis_client, source, deleted_raw_ids)
-            if reparented_nodes:
-                reparented_cache_entries = [
-                    HierarchyNodeCacheEntry.from_db_model(node)
-                    for node in reparented_nodes
-                ]
-                cache_hierarchy_nodes_batch(
-                    redis_client, source, reparented_cache_entries
-                )
-            if stale_removed or deleted_raw_ids or reparented_nodes:
-                task_logger.info(
-                    f"Hierarchy node pruning: cc_pair={cc_pair_id} "
-                    f"stale_entries_removed={stale_removed} "
-                    f"nodes_deleted={len(deleted_raw_ids)} "
-                    f"nodes_reparented={len(reparented_nodes)}"
-                )
     except Exception as e:
         task_logger.exception(
             f"Pruning exceptioned: cc_pair={cc_pair_id} "

backend/onyx/background/celery/tasks/user_file_processing/tasks.py

@@ -12,9 +12,9 @@ from redis import Redis
 from redis.lock import Lock as RedisLock
 from retry import retry
 from sqlalchemy import select
+from sqlalchemy.orm import selectinload
 from sqlalchemy.orm import Session
 
-from onyx.access.access import build_access_for_user_files
 from onyx.background.celery.apps.app_base import task_logger
 from onyx.background.celery.celery_redis import celery_get_queue_length
 from onyx.background.celery.celery_utils import httpx_init_vespa_pool
@@ -43,9 +43,7 @@ from onyx.db.enums import UserFileStatus
 from onyx.db.models import UserFile
 from onyx.db.search_settings import get_active_search_settings
 from onyx.db.search_settings import get_active_search_settings_list
-from onyx.db.user_file import fetch_user_files_with_access_relationships
 from onyx.document_index.factory import get_all_document_indices
-from onyx.document_index.interfaces import VespaDocumentFields
 from onyx.document_index.interfaces import VespaDocumentUserFields
 from onyx.document_index.vespa_constants import DOCUMENT_ID_ENDPOINT
 from onyx.file_store.file_store import get_default_file_store
@@ -56,7 +54,6 @@ from onyx.indexing.adapters.user_file_indexing_adapter import UserFileIndexingAd
 from onyx.indexing.embedder import DefaultIndexingEmbedder
 from onyx.indexing.indexing_pipeline import run_indexing_pipeline
 from onyx.redis.redis_pool import get_redis_client
-from onyx.utils.variable_functionality import global_version
 
 
 def _as_uuid(value: str | UUID) -> UUID:
@@ -523,7 +520,6 @@ def process_user_file_impl(
         task_logger.exception(
             f"process_user_file_impl - Error processing file id={user_file_id} - {e.__class__.__name__}"
         )
-        raise
     finally:
         if file_lock is not None and file_lock.owned():
             file_lock.release()
@@ -679,7 +675,6 @@ def delete_user_file_impl(
         task_logger.exception(
             f"delete_user_file_impl - Error processing file id={user_file_id} - {e.__class__.__name__}"
         )
-        raise
     finally:
         if file_lock is not None and file_lock.owned():
             file_lock.release()
@@ -794,12 +789,11 @@ def project_sync_user_file_impl(
 
     try:
         with get_session_with_current_tenant() as db_session:
-            user_files = fetch_user_files_with_access_relationships(
-                [user_file_id],
-                db_session,
-                eager_load_groups=global_version.is_ee_version(),
-            )
-            user_file = user_files[0] if user_files else None
+            user_file = db_session.execute(
+                select(UserFile)
+                .where(UserFile.id == _as_uuid(user_file_id))
+                .options(selectinload(UserFile.assistants))
+            ).scalar_one_or_none()
             if not user_file:
                 task_logger.info(
                     f"project_sync_user_file_impl - User file not found id={user_file_id}"
@@ -827,21 +821,12 @@ def project_sync_user_file_impl(
 
                 project_ids = [project.id for project in user_file.projects]
                 persona_ids = [p.id for p in user_file.assistants if not p.deleted]
-
-                file_id_str = str(user_file.id)
-                access_map = build_access_for_user_files([user_file])
-                access = access_map.get(file_id_str)
-
                 for retry_document_index in retry_document_indices:
                     retry_document_index.update_single(
-                        doc_id=file_id_str,
+                        doc_id=str(user_file.id),
                         tenant_id=tenant_id,
                         chunk_count=user_file.chunk_count,
-                        fields=(
-                            VespaDocumentFields(access=access)
-                            if access is not None
-                            else None
-                        ),
+                        fields=None,
                         user_fields=VespaDocumentUserFields(
                             user_projects=project_ids,
                             personas=persona_ids,
@@ -864,7 +849,6 @@ def project_sync_user_file_impl(
         task_logger.exception(
             f"project_sync_user_file_impl - Error syncing project for file id={user_file_id} - {e.__class__.__name__}"
         )
-        raise
     finally:
         if file_lock is not None and file_lock.owned():
             file_lock.release()

backend/onyx/background/celery/versioned_apps/background.py

@@ -0,0 +1,10 @@
+from celery import Celery
+
+from onyx.utils.variable_functionality import fetch_versioned_implementation
+from onyx.utils.variable_functionality import set_is_ee_based_on_env_variable
+
+set_is_ee_based_on_env_variable()
+app: Celery = fetch_versioned_implementation(
+    "onyx.background.celery.apps.background",
+    "celery_app",
+)

backend/onyx/background/indexing/run_docfetching.py

@@ -45,7 +45,6 @@ from onyx.db.enums import ConnectorCredentialPairStatus
 from onyx.db.enums import IndexingStatus
 from onyx.db.enums import IndexModelStatus
 from onyx.db.enums import ProcessingMode
-from onyx.db.hierarchy import upsert_hierarchy_node_cc_pair_entries
 from onyx.db.hierarchy import upsert_hierarchy_nodes_batch
 from onyx.db.index_attempt import create_index_attempt_error
 from onyx.db.index_attempt import get_index_attempt
@@ -59,6 +58,8 @@ from onyx.file_store.document_batch_storage import DocumentBatchStorage
 from onyx.file_store.document_batch_storage import get_document_batch_storage
 from onyx.indexing.indexing_heartbeat import IndexingHeartbeatInterface
 from onyx.indexing.indexing_pipeline import index_doc_batch_prepare
+from onyx.indexing.postgres_sanitization import sanitize_document_for_postgres
+from onyx.indexing.postgres_sanitization import sanitize_hierarchy_nodes_for_postgres
 from onyx.redis.redis_hierarchy import cache_hierarchy_nodes_batch
 from onyx.redis.redis_hierarchy import ensure_source_node_exists
 from onyx.redis.redis_hierarchy import get_node_id_from_raw_id
@@ -70,8 +71,6 @@ from onyx.server.features.build.indexing.persistent_document_writer import (
 )
 from onyx.utils.logger import setup_logger
 from onyx.utils.middleware import make_randomized_onyx_request_id
-from onyx.utils.postgres_sanitization import sanitize_document_for_postgres
-from onyx.utils.postgres_sanitization import sanitize_hierarchy_nodes_for_postgres
 from onyx.utils.variable_functionality import global_version
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.contextvars import INDEX_ATTEMPT_INFO_CONTEXTVAR
@@ -588,14 +587,6 @@ def connector_document_extraction(
                             is_connector_public=is_connector_public,
                         )
 
-                        upsert_hierarchy_node_cc_pair_entries(
-                            db_session=db_session,
-                            hierarchy_node_ids=[n.id for n in upserted_nodes],
-                            connector_id=db_connector.id,
-                            credential_id=db_credential.id,
-                            commit=True,
-                        )
-
                         # Cache in Redis for fast ancestor resolution during doc processing
                         redis_client = get_redis_client(tenant_id=tenant_id)
                         cache_entries = [

backend/onyx/background/periodic_poller.py

@@ -59,12 +59,6 @@ def _run_auto_llm_update() -> None:
         sync_llm_models_from_github(db_session)
 
 
-def _run_cache_cleanup() -> None:
-    from onyx.cache.postgres_backend import cleanup_expired_cache_entries
-
-    cleanup_expired_cache_entries()
-
-
 def _run_scheduled_eval() -> None:
     from onyx.configs.app_configs import BRAINTRUST_API_KEY
     from onyx.configs.app_configs import SCHEDULED_EVAL_DATASET_NAMES
@@ -106,26 +100,12 @@ def _run_scheduled_eval() -> None:
             )
 
 
-_CACHE_CLEANUP_INTERVAL_SECONDS = 300
-
-
 def _build_periodic_tasks() -> list[_PeriodicTaskDef]:
-    from onyx.cache.interface import CacheBackendType
     from onyx.configs.app_configs import AUTO_LLM_CONFIG_URL
     from onyx.configs.app_configs import AUTO_LLM_UPDATE_INTERVAL_SECONDS
-    from onyx.configs.app_configs import CACHE_BACKEND
     from onyx.configs.app_configs import SCHEDULED_EVAL_DATASET_NAMES
 
     tasks: list[_PeriodicTaskDef] = []
-    if CACHE_BACKEND == CacheBackendType.POSTGRES:
-        tasks.append(
-            _PeriodicTaskDef(
-                name="cache-cleanup",
-                interval_seconds=_CACHE_CLEANUP_INTERVAL_SECONDS,
-                lock_id=PERIODIC_TASK_LOCK_BASE + 2,
-                run_fn=_run_cache_cleanup,
-            )
-        )
     if AUTO_LLM_CONFIG_URL:
         tasks.append(
             _PeriodicTaskDef(

backend/onyx/background/task_utils.py

@@ -75,41 +75,31 @@ def _claim_next_processing_file(db_session: Session) -> UUID | None:
     return file_id
 
 
-def _claim_next_deleting_file(
-    db_session: Session,
-    exclude_ids: set[UUID] | None = None,
-) -> UUID | None:
+def _claim_next_deleting_file(db_session: Session) -> UUID | None:
     """Claim the next DELETING file.
 
     No status transition needed — the impl deletes the row on success.
     The short-lived FOR UPDATE lock prevents concurrent claims.
-    *exclude_ids* prevents re-processing the same file if the impl fails.
     """
-    stmt = (
+    file_id = db_session.execute(
         select(UserFile.id)
         .where(UserFile.status == UserFileStatus.DELETING)
         .order_by(UserFile.created_at)
         .limit(1)
         .with_for_update(skip_locked=True)
-    )
-    if exclude_ids:
-        stmt = stmt.where(UserFile.id.notin_(exclude_ids))
-    file_id = db_session.execute(stmt).scalar_one_or_none()
+    ).scalar_one_or_none()
+    # Commit to release the row lock promptly.
     db_session.commit()
     return file_id
 
 
-def _claim_next_sync_file(
-    db_session: Session,
-    exclude_ids: set[UUID] | None = None,
-) -> UUID | None:
+def _claim_next_sync_file(db_session: Session) -> UUID | None:
     """Claim the next file needing project/persona sync.
 
     No status transition needed — the impl clears the sync flags on
     success.  The short-lived FOR UPDATE lock prevents concurrent claims.
-    *exclude_ids* prevents re-processing the same file if the impl fails.
     """
-    stmt = (
+    file_id = db_session.execute(
         select(UserFile.id)
         .where(
             sa.and_(
@@ -123,10 +113,7 @@ def _claim_next_sync_file(
         .order_by(UserFile.created_at)
         .limit(1)
         .with_for_update(skip_locked=True)
-    )
-    if exclude_ids:
-        stmt = stmt.where(UserFile.id.notin_(exclude_ids))
-    file_id = db_session.execute(stmt).scalar_one_or_none()
+    ).scalar_one_or_none()
     db_session.commit()
     return file_id
 
@@ -148,14 +135,11 @@ def drain_processing_loop(tenant_id: str) -> None:
             file_id = _claim_next_processing_file(session)
         if file_id is None:
             break
-        try:
-            process_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to process user file {file_id}")
+        process_user_file_impl(
+            user_file_id=str(file_id),
+            tenant_id=tenant_id,
+            redis_locking=False,
+        )
 
 
 def drain_delete_loop(tenant_id: str) -> None:
@@ -165,21 +149,16 @@ def drain_delete_loop(tenant_id: str) -> None:
     )
     from onyx.db.engine.sql_engine import get_session_with_current_tenant
 
-    failed: set[UUID] = set()
     while True:
         with get_session_with_current_tenant() as session:
-            file_id = _claim_next_deleting_file(session, exclude_ids=failed)
+            file_id = _claim_next_deleting_file(session)
         if file_id is None:
             break
-        try:
-            delete_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to delete user file {file_id}")
-            failed.add(file_id)
+        delete_user_file_impl(
+            user_file_id=str(file_id),
+            tenant_id=tenant_id,
+            redis_locking=False,
+        )
 
 
 def drain_project_sync_loop(tenant_id: str) -> None:
@@ -189,18 +168,13 @@ def drain_project_sync_loop(tenant_id: str) -> None:
     )
     from onyx.db.engine.sql_engine import get_session_with_current_tenant
 
-    failed: set[UUID] = set()
     while True:
         with get_session_with_current_tenant() as session:
-            file_id = _claim_next_sync_file(session, exclude_ids=failed)
+            file_id = _claim_next_sync_file(session)
         if file_id is None:
             break
-        try:
-            project_sync_user_file_impl(
-                user_file_id=str(file_id),
-                tenant_id=tenant_id,
-                redis_locking=False,
-            )
-        except Exception:
-            logger.exception(f"Failed to sync user file {file_id}")
-            failed.add(file_id)
+        project_sync_user_file_impl(
+            user_file_id=str(file_id),
+            tenant_id=tenant_id,
+            redis_locking=False,
+        )

backend/onyx/cache/factory.py

@@ -12,15 +12,9 @@ def _build_redis_backend(tenant_id: str) -> CacheBackend:
     return RedisCacheBackend(redis_pool.get_client(tenant_id))
 
 
-def _build_postgres_backend(tenant_id: str) -> CacheBackend:
-    from onyx.cache.postgres_backend import PostgresCacheBackend
-
-    return PostgresCacheBackend(tenant_id)
-
-
 _BACKEND_BUILDERS: dict[CacheBackendType, Callable[[str], CacheBackend]] = {
     CacheBackendType.REDIS: _build_redis_backend,
-    CacheBackendType.POSTGRES: _build_postgres_backend,
+    # CacheBackendType.POSTGRES will be added in a follow-up PR.
 }
 
 

backend/onyx/cache/interface.py

@@ -1,20 +1,6 @@
 import abc
 from enum import Enum
 
-from redis.exceptions import RedisError
-from sqlalchemy.exc import SQLAlchemyError
-
-TTL_KEY_NOT_FOUND = -2
-TTL_NO_EXPIRY = -1
-
-CACHE_TRANSIENT_ERRORS: tuple[type[Exception], ...] = (RedisError, SQLAlchemyError)
-"""Exception types that represent transient cache connectivity / operational
-failures.  Callers that want to fail-open (or fail-closed) on cache errors
-should catch this tuple instead of bare ``Exception``.
-
-When adding a new ``CacheBackend`` implementation, add its transient error
-base class(es) here so all call-sites pick it up automatically."""
-
 
 class CacheBackendType(str, Enum):
     REDIS = "redis"
@@ -40,14 +26,6 @@ class CacheLock(abc.ABC):
     def owned(self) -> bool:
         raise NotImplementedError
 
-    def __enter__(self) -> "CacheLock":
-        if not self.acquire():
-            raise RuntimeError("Failed to acquire lock")
-        return self
-
-    def __exit__(self, *args: object) -> None:
-        self.release()
-
 
 class CacheBackend(abc.ABC):
     """Thin abstraction over a key-value cache with TTL, locks, and blocking lists.
@@ -87,11 +65,7 @@ class CacheBackend(abc.ABC):
 
     @abc.abstractmethod
     def ttl(self, key: str) -> int:
-        """Return remaining TTL in seconds.
-
-        Returns ``TTL_NO_EXPIRY`` (-1) if key exists without expiry,
-        ``TTL_KEY_NOT_FOUND`` (-2) if key is missing or expired.
-        """
+        """Return remaining TTL in seconds. -1 if no expiry, -2 if key missing."""
         raise NotImplementedError
 
     # -- distributed lock --------------------------------------------------

backend/onyx/cache/postgres_backend.py

@@ -1,323 +0,0 @@
-"""PostgreSQL-backed ``CacheBackend`` for NO_VECTOR_DB deployments.
-
-Uses the ``cache_store`` table for key-value storage, PostgreSQL advisory locks
-for distributed locking, and a polling loop for the BLPOP pattern.
-"""
-
-import hashlib
-import struct
-import time
-import uuid
-from contextlib import AbstractContextManager
-from datetime import datetime
-from datetime import timedelta
-from datetime import timezone
-
-from sqlalchemy import delete
-from sqlalchemy import func
-from sqlalchemy import or_
-from sqlalchemy import select
-from sqlalchemy import update
-from sqlalchemy.dialects.postgresql import insert as pg_insert
-from sqlalchemy.orm import Session
-
-from onyx.cache.interface import CacheBackend
-from onyx.cache.interface import CacheLock
-from onyx.cache.interface import TTL_KEY_NOT_FOUND
-from onyx.cache.interface import TTL_NO_EXPIRY
-from onyx.db.models import CacheStore
-
-_LIST_KEY_PREFIX = "_q:"
-# ASCII: ':' (0x3A) < ';' (0x3B). Upper bound for range queries so [prefix+, prefix;)
-# captures all list-item keys (e.g. _q:mylist:123:uuid) without including other
-# lists whose names share a prefix (e.g. _q:mylist2:...).
-_LIST_KEY_RANGE_TERMINATOR = ";"
-_LIST_ITEM_TTL_SECONDS = 3600
-_LOCK_POLL_INTERVAL = 0.1
-_BLPOP_POLL_INTERVAL = 0.25
-
-
-def _list_item_key(key: str) -> str:
-    """Unique key for a list item. Timestamp for FIFO ordering; UUID prevents
-    collision when concurrent rpush calls occur within the same nanosecond.
-    """
-    return f"{_LIST_KEY_PREFIX}{key}:{time.time_ns()}:{uuid.uuid4().hex}"
-
-
-def _to_bytes(value: str | bytes | int | float) -> bytes:
-    if isinstance(value, bytes):
-        return value
-    return str(value).encode()
-
-
-# ------------------------------------------------------------------
-# Lock
-# ------------------------------------------------------------------
-
-
-class PostgresCacheLock(CacheLock):
-    """Advisory-lock-based distributed lock.
-
-    Uses ``get_session_with_tenant`` for connection lifecycle.  The lock is tied
-    to the session's connection; releasing or closing the session frees it.
-
-    NOTE: Unlike Redis locks, advisory locks do not auto-expire after
-    ``timeout`` seconds.  They are released when ``release()`` is
-    called or when the session is closed.
-    """
-
-    def __init__(self, lock_id: int, timeout: float | None, tenant_id: str) -> None:
-        self._lock_id = lock_id
-        self._timeout = timeout
-        self._tenant_id = tenant_id
-        self._session_cm: AbstractContextManager[Session] | None = None
-        self._session: Session | None = None
-        self._acquired = False
-
-    def acquire(
-        self,
-        blocking: bool = True,
-        blocking_timeout: float | None = None,
-    ) -> bool:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        self._session_cm = get_session_with_tenant(tenant_id=self._tenant_id)
-        self._session = self._session_cm.__enter__()
-        try:
-            if not blocking:
-                return self._try_lock()
-
-            effective_timeout = blocking_timeout or self._timeout
-            deadline = (
-                (time.monotonic() + effective_timeout) if effective_timeout else None
-            )
-            while True:
-                if self._try_lock():
-                    return True
-                if deadline is not None and time.monotonic() >= deadline:
-                    return False
-                time.sleep(_LOCK_POLL_INTERVAL)
-        finally:
-            if not self._acquired:
-                self._close_session()
-
-    def release(self) -> None:
-        if not self._acquired or self._session is None:
-            return
-        try:
-            self._session.execute(select(func.pg_advisory_unlock(self._lock_id)))
-        finally:
-            self._acquired = False
-            self._close_session()
-
-    def owned(self) -> bool:
-        return self._acquired
-
-    def _close_session(self) -> None:
-        if self._session_cm is not None:
-            try:
-                self._session_cm.__exit__(None, None, None)
-            finally:
-                self._session_cm = None
-                self._session = None
-
-    def _try_lock(self) -> bool:
-        assert self._session is not None
-        result = self._session.execute(
-            select(func.pg_try_advisory_lock(self._lock_id))
-        ).scalar()
-        if result:
-            self._acquired = True
-            return True
-        return False
-
-
-# ------------------------------------------------------------------
-# Backend
-# ------------------------------------------------------------------
-
-
-class PostgresCacheBackend(CacheBackend):
-    """``CacheBackend`` backed by the ``cache_store`` table in PostgreSQL.
-
-    Each operation opens and closes its own database session so the backend
-    is safe to share across threads.  Tenant isolation is handled by
-    SQLAlchemy's ``schema_translate_map`` (set by ``get_session_with_tenant``).
-    """
-
-    def __init__(self, tenant_id: str) -> None:
-        self._tenant_id = tenant_id
-
-    # -- basic key/value ---------------------------------------------------
-
-    def get(self, key: str) -> bytes | None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = select(CacheStore.value).where(
-            CacheStore.key == key,
-            or_(CacheStore.expires_at.is_(None), CacheStore.expires_at > func.now()),
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            value = session.execute(stmt).scalar_one_or_none()
-        if value is None:
-            return None
-        return bytes(value)
-
-    def set(
-        self,
-        key: str,
-        value: str | bytes | int | float,
-        ex: int | None = None,
-    ) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        value_bytes = _to_bytes(value)
-        expires_at = (
-            datetime.now(timezone.utc) + timedelta(seconds=ex)
-            if ex is not None
-            else None
-        )
-        stmt = (
-            pg_insert(CacheStore)
-            .values(key=key, value=value_bytes, expires_at=expires_at)
-            .on_conflict_do_update(
-                index_elements=[CacheStore.key],
-                set_={"value": value_bytes, "expires_at": expires_at},
-            )
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(stmt)
-            session.commit()
-
-    def delete(self, key: str) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(delete(CacheStore).where(CacheStore.key == key))
-            session.commit()
-
-    def exists(self, key: str) -> bool:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = (
-            select(CacheStore.key)
-            .where(
-                CacheStore.key == key,
-                or_(
-                    CacheStore.expires_at.is_(None),
-                    CacheStore.expires_at > func.now(),
-                ),
-            )
-            .limit(1)
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            return session.execute(stmt).first() is not None
-
-    # -- TTL ---------------------------------------------------------------
-
-    def expire(self, key: str, seconds: int) -> None:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        new_exp = datetime.now(timezone.utc) + timedelta(seconds=seconds)
-        stmt = (
-            update(CacheStore).where(CacheStore.key == key).values(expires_at=new_exp)
-        )
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            session.execute(stmt)
-            session.commit()
-
-    def ttl(self, key: str) -> int:
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        stmt = select(CacheStore.expires_at).where(CacheStore.key == key)
-        with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-            result = session.execute(stmt).first()
-        if result is None:
-            return TTL_KEY_NOT_FOUND
-        expires_at: datetime | None = result[0]
-        if expires_at is None:
-            return TTL_NO_EXPIRY
-        remaining = (expires_at - datetime.now(timezone.utc)).total_seconds()
-        if remaining <= 0:
-            return TTL_KEY_NOT_FOUND
-        return int(remaining)
-
-    # -- distributed lock --------------------------------------------------
-
-    def lock(self, name: str, timeout: float | None = None) -> CacheLock:
-        return PostgresCacheLock(
-            self._lock_id_for(name), timeout, tenant_id=self._tenant_id
-        )
-
-    # -- blocking list (MCP OAuth BLPOP pattern) ---------------------------
-
-    def rpush(self, key: str, value: str | bytes) -> None:
-        self.set(_list_item_key(key), value, ex=_LIST_ITEM_TTL_SECONDS)
-
-    def blpop(self, keys: list[str], timeout: int = 0) -> tuple[bytes, bytes] | None:
-        if timeout <= 0:
-            raise ValueError(
-                "PostgresCacheBackend.blpop requires timeout > 0. "
-                "timeout=0 would block the calling thread indefinitely "
-                "with no way to interrupt short of process termination."
-            )
-        from onyx.db.engine.sql_engine import get_session_with_tenant
-
-        deadline = time.monotonic() + timeout
-        while True:
-            for key in keys:
-                lower = f"{_LIST_KEY_PREFIX}{key}:"
-                upper = f"{_LIST_KEY_PREFIX}{key}{_LIST_KEY_RANGE_TERMINATOR}"
-                stmt = (
-                    select(CacheStore)
-                    .where(
-                        CacheStore.key >= lower,
-                        CacheStore.key < upper,
-                        or_(
-                            CacheStore.expires_at.is_(None),
-                            CacheStore.expires_at > func.now(),
-                        ),
-                    )
-                    .order_by(CacheStore.key)
-                    .limit(1)
-                    .with_for_update(skip_locked=True)
-                )
-                with get_session_with_tenant(tenant_id=self._tenant_id) as session:
-                    row = session.execute(stmt).scalars().first()
-                    if row is not None:
-                        value = bytes(row.value) if row.value else b""
-                        session.delete(row)
-                        session.commit()
-                        return (key.encode(), value)
-            if time.monotonic() >= deadline:
-                return None
-            time.sleep(_BLPOP_POLL_INTERVAL)
-
-    # -- helpers -----------------------------------------------------------
-
-    def _lock_id_for(self, name: str) -> int:
-        """Map *name* to a 64-bit signed int for ``pg_advisory_lock``."""
-        h = hashlib.md5(f"{self._tenant_id}:{name}".encode()).digest()
-        return struct.unpack("q", h[:8])[0]
-
-
-# ------------------------------------------------------------------
-# Periodic cleanup
-# ------------------------------------------------------------------
-
-
-def cleanup_expired_cache_entries() -> None:
-    """Delete rows whose ``expires_at`` is in the past.
-
-    Called by the periodic poller every 5 minutes.
-    """
-    from onyx.db.engine.sql_engine import get_session_with_current_tenant
-
-    with get_session_with_current_tenant() as session:
-        session.execute(
-            delete(CacheStore).where(
-                CacheStore.expires_at.is_not(None),
-                CacheStore.expires_at < func.now(),
-            )
-        )
-        session.commit()

backend/onyx/chat/chat_processing_checker.py

@@ -1,52 +1,57 @@
 from uuid import UUID
 
-from onyx.cache.interface import CacheBackend
+from redis.client import Redis
 
+# Redis key prefixes for chat message processing
 PREFIX = "chatprocessing"
 FENCE_PREFIX = f"{PREFIX}_fence"
 FENCE_TTL = 30 * 60  # 30 minutes
 
 
 def _get_fence_key(chat_session_id: UUID) -> str:
-    """Generate the cache key for a chat session processing fence.
+    """
+    Generate the Redis key for a chat session processing a message.
 
     Args:
         chat_session_id: The UUID of the chat session
 
     Returns:
-        The fence key string. Tenant isolation is handled automatically
-        by the cache backend (Redis key-prefixing or Postgres schema routing).
+        The fence key string (tenant_id is automatically added by the Redis client)
     """
     return f"{FENCE_PREFIX}_{chat_session_id}"
 
 
 def set_processing_status(
-    chat_session_id: UUID, cache: CacheBackend, value: bool
+    chat_session_id: UUID, redis_client: Redis, value: bool
 ) -> None:
-    """Set or clear the fence for a chat session processing a message.
+    """
+    Set or clear the fence for a chat session processing a message.
 
-    If the key exists, a message is being processed.
+    If the key exists, we are processing a message. If the key does not exist, we are not processing a message.
 
     Args:
         chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: The Redis client to use
         value: True to set the fence, False to clear it
     """
     fence_key = _get_fence_key(chat_session_id)
+
     if value:
-        cache.set(fence_key, 0, ex=FENCE_TTL)
+        redis_client.set(fence_key, 0, ex=FENCE_TTL)
     else:
-        cache.delete(fence_key)
+        redis_client.delete(fence_key)
 
 
-def is_chat_session_processing(chat_session_id: UUID, cache: CacheBackend) -> bool:
-    """Check if the chat session is processing a message.
+def is_chat_session_processing(chat_session_id: UUID, redis_client: Redis) -> bool:
+    """
+    Check if the chat session is processing a message.
 
     Args:
         chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: The Redis client to use
 
     Returns:
         True if the chat session is processing a message, False otherwise
     """
-    return cache.exists(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    return bool(redis_client.exists(fence_key))

backend/onyx/chat/llm_loop.py

@@ -36,6 +36,7 @@ from onyx.db.memory import add_memory
 from onyx.db.memory import update_memory_at_index
 from onyx.db.memory import UserMemoryContext
 from onyx.db.models import Persona
+from onyx.llm.constants import LlmProviderNames
 from onyx.llm.interfaces import LLM
 from onyx.llm.interfaces import LLMUserIdentity
 from onyx.llm.interfaces import ToolChoiceOptions
@@ -50,9 +51,7 @@ from onyx.tools.built_in_tools import CITEABLE_TOOLS_NAMES
 from onyx.tools.built_in_tools import STOPPING_TOOLS_NAMES
 from onyx.tools.interface import Tool
 from onyx.tools.models import ChatFile
-from onyx.tools.models import CustomToolCallSummary
 from onyx.tools.models import MemoryToolResponseSnapshot
-from onyx.tools.models import PythonToolRichResponse
 from onyx.tools.models import ToolCallInfo
 from onyx.tools.models import ToolCallKickoff
 from onyx.tools.models import ToolResponse
@@ -84,6 +83,28 @@ def _looks_like_xml_tool_call_payload(text: str | None) -> bool:
     )
 
 
+def _should_keep_bedrock_tool_definitions(
+    llm: object, simple_chat_history: list[ChatMessageSimple]
+) -> bool:
+    """Bedrock requires tool config when history includes toolUse/toolResult blocks."""
+    model_provider = getattr(getattr(llm, "config", None), "model_provider", None)
+    if model_provider not in {
+        LlmProviderNames.BEDROCK,
+        LlmProviderNames.BEDROCK_CONVERSE,
+    }:
+        return False
+
+    return any(
+        (
+            msg.message_type == MessageType.ASSISTANT
+            and msg.tool_calls
+            and len(msg.tool_calls) > 0
+        )
+        or msg.message_type == MessageType.TOOL_CALL_RESPONSE
+        for msg in simple_chat_history
+    )
+
+
 def _try_fallback_tool_extraction(
     llm_step_result: LlmStepResult,
     tool_choice: ToolChoiceOptions,
@@ -664,7 +685,12 @@ def run_llm_loop(
             elif out_of_cycles or ran_image_gen:
                 # Last cycle, no tools allowed, just answer!
                 tool_choice = ToolChoiceOptions.NONE
-                final_tools = []
+                # Bedrock requires tool config in requests that include toolUse/toolResult history.
+                final_tools = (
+                    tools
+                    if _should_keep_bedrock_tool_definitions(llm, simple_chat_history)
+                    else []
+                )
             else:
                 tool_choice = ToolChoiceOptions.AUTO
                 final_tools = tools
@@ -940,13 +966,6 @@ def run_llm_loop(
                 ):
                     generated_images = tool_response.rich_response.generated_images
 
-                # Extract generated_files if this is a code interpreter response
-                generated_files = None
-                if isinstance(tool_response.rich_response, PythonToolRichResponse):
-                    generated_files = (
-                        tool_response.rich_response.generated_files or None
-                    )
-
                 # Persist memory if this is a memory tool response
                 memory_snapshot: MemoryToolResponseSnapshot | None = None
                 if isinstance(tool_response.rich_response, MemoryToolResponse):
@@ -981,10 +1000,6 @@ def run_llm_loop(
 
                 if memory_snapshot:
                     saved_response = json.dumps(memory_snapshot.model_dump())
-                elif isinstance(tool_response.rich_response, CustomToolCallSummary):
-                    saved_response = json.dumps(
-                        tool_response.rich_response.model_dump()
-                    )
                 elif isinstance(tool_response.rich_response, str):
                     saved_response = tool_response.rich_response
                 else:
@@ -1002,7 +1017,6 @@ def run_llm_loop(
                     tool_call_response=saved_response,
                     search_docs=displayed_docs or search_docs,
                     generated_images=generated_images,
-                    generated_files=generated_files,
                 )
                 # Add to state container for partial save support
                 state_container.add_tool_call(tool_call_info)

backend/onyx/chat/llm_step.py

@@ -15,7 +15,6 @@ from onyx.chat.citation_processor import DynamicCitationProcessor
 from onyx.chat.emitter import Emitter
 from onyx.chat.models import ChatMessageSimple
 from onyx.chat.models import LlmStepResult
-from onyx.chat.tool_call_args_streaming import maybe_emit_argument_delta
 from onyx.configs.app_configs import LOG_ONYX_MODEL_INTERACTIONS
 from onyx.configs.app_configs import PROMPT_CACHE_CHAT_HISTORY
 from onyx.configs.constants import MessageType
@@ -55,9 +54,7 @@ from onyx.server.query_and_chat.streaming_models import ReasoningStart
 from onyx.tools.models import ToolCallKickoff
 from onyx.tracing.framework.create import generation_span
 from onyx.utils.b64 import get_image_type_from_bytes
-from onyx.utils.jsonriver import Parser
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string
 from onyx.utils.text_processing import find_all_json_objects
 
 logger = setup_logger()
@@ -169,6 +166,15 @@ def _find_function_calls_open_marker(text_lower: str) -> int:
         search_from = idx + 1
 
 
+def _sanitize_llm_output(value: str) -> str:
+    """Remove characters that PostgreSQL's text/JSONB types cannot store.
+
+    - NULL bytes (\x00): Not allowed in PostgreSQL text types
+    - UTF-16 surrogates (\ud800-\udfff): Invalid in UTF-8 encoding
+    """
+    return "".join(c for c in value if c != "\x00" and not ("\ud800" <= c <= "\udfff"))
+
+
 def _try_parse_json_string(value: Any) -> Any:
     """Attempt to parse a JSON string value into its Python equivalent.
 
@@ -216,7 +222,9 @@ def _parse_tool_args_to_dict(raw_args: Any) -> dict[str, Any]:
     if isinstance(raw_args, dict):
         # Parse any string values that look like JSON arrays/objects
         return {
-            k: _try_parse_json_string(sanitize_string(v) if isinstance(v, str) else v)
+            k: _try_parse_json_string(
+                _sanitize_llm_output(v) if isinstance(v, str) else v
+            )
             for k, v in raw_args.items()
         }
 
@@ -224,7 +232,7 @@ def _parse_tool_args_to_dict(raw_args: Any) -> dict[str, Any]:
         return {}
 
     # Sanitize before parsing to remove NULL bytes and surrogates
-    raw_args = sanitize_string(raw_args)
+    raw_args = _sanitize_llm_output(raw_args)
 
     try:
         parsed1: Any = json.loads(raw_args)
@@ -537,12 +545,12 @@ def _extract_xml_attribute(attrs: str, attr_name: str) -> str | None:
     )
     if not attr_match:
         return None
-    return sanitize_string(unescape(attr_match.group(2).strip()))
+    return _sanitize_llm_output(unescape(attr_match.group(2).strip()))
 
 
 def _parse_xml_parameter_value(raw_value: str, string_attr: str | None) -> Any:
     """Parse a parameter value from XML-style tool call payloads."""
-    value = sanitize_string(unescape(raw_value).strip())
+    value = _sanitize_llm_output(unescape(raw_value).strip())
 
     if string_attr and string_attr.lower() == "true":
         return value
@@ -561,7 +569,6 @@ def _resolve_tool_arguments(obj: dict[str, Any]) -> dict[str, Any] | None:
     """
     arguments = obj.get("arguments", obj.get("parameters", {}))
     if isinstance(arguments, str):
-        arguments = sanitize_string(arguments)
         try:
             arguments = json.loads(arguments)
         except json.JSONDecodeError:
@@ -1011,7 +1018,6 @@ def run_llm_step_pkt_generator(
         )
 
     id_to_tool_call_map: dict[int, dict[str, Any]] = {}
-    arg_parsers: dict[int, Parser] = {}
     reasoning_start = False
     answer_start = False
     accumulated_reasoning = ""
@@ -1218,14 +1224,7 @@ def run_llm_step_pkt_generator(
                 yield from _close_reasoning_if_active()
 
                 for tool_call_delta in delta.tool_calls:
-                    # maybe_emit depends and update being called first and attaching the delta
                     _update_tool_call_with_delta(id_to_tool_call_map, tool_call_delta)
-                    yield from maybe_emit_argument_delta(
-                        tool_calls_in_progress=id_to_tool_call_map,
-                        tool_call_delta=tool_call_delta,
-                        placement=_current_placement(),
-                        parsers=arg_parsers,
-                    )
 
         # Flush any tail text buffered while checking for split "<function_calls" markers.
         filtered_content_tail = xml_tool_call_content_filter.flush()

backend/onyx/chat/process_message.py

@@ -11,10 +11,9 @@ from contextvars import Token
 from uuid import UUID
 
 from pydantic import BaseModel
+from redis.client import Redis
 from sqlalchemy.orm import Session
 
-from onyx.cache.factory import get_cache_backend
-from onyx.cache.interface import CacheBackend
 from onyx.chat.chat_processing_checker import set_processing_status
 from onyx.chat.chat_state import ChatStateContainer
 from onyx.chat.chat_state import run_chat_loop_with_state_containers
@@ -80,6 +79,7 @@ from onyx.llm.request_context import reset_llm_mock_response
 from onyx.llm.request_context import set_llm_mock_response
 from onyx.llm.utils import litellm_exception_to_error_msg
 from onyx.onyxbot.slack.models import SlackContext
+from onyx.redis.redis_pool import get_redis_client
 from onyx.server.query_and_chat.models import AUTO_PLACE_AFTER_LATEST_MESSAGE
 from onyx.server.query_and_chat.models import MessageResponseIDInfo
 from onyx.server.query_and_chat.models import SendMessageRequest
@@ -448,7 +448,7 @@ def handle_stream_message_objects(
 
     llm: LLM | None = None
     chat_session: ChatSession | None = None
-    cache: CacheBackend | None = None
+    redis_client: Redis | None = None
 
     user_id = user.id
     if user.is_anonymous:
@@ -809,19 +809,19 @@ def handle_stream_message_objects(
             )
             simple_chat_history.insert(0, summary_simple)
 
-        cache = get_cache_backend()
+        redis_client = get_redis_client()
 
         reset_cancel_status(
             chat_session.id,
-            cache,
+            redis_client,
         )
 
         def check_is_connected() -> bool:
-            return check_stop_signal(chat_session.id, cache)
+            return check_stop_signal(chat_session.id, redis_client)
 
         set_processing_status(
             chat_session_id=chat_session.id,
-            cache=cache,
+            redis_client=redis_client,
             value=True,
         )
 
@@ -968,10 +968,10 @@ def handle_stream_message_objects(
             reset_llm_mock_response(mock_response_token)
 
         try:
-            if cache is not None and chat_session is not None:
+            if redis_client is not None and chat_session is not None:
                 set_processing_status(
                     chat_session_id=chat_session.id,
-                    cache=cache,
+                    redis_client=redis_client,
                     value=False,
                 )
         except Exception:

backend/onyx/chat/save_chat.py

@@ -1,5 +1,4 @@
 import json
-import mimetypes
 
 from sqlalchemy.orm import Session
 
@@ -13,42 +12,14 @@ from onyx.db.chat import create_db_search_doc
 from onyx.db.models import ChatMessage
 from onyx.db.models import ToolCall
 from onyx.db.tools import create_tool_call_no_commit
-from onyx.file_store.models import FileDescriptor
 from onyx.natural_language_processing.utils import BaseTokenizer
 from onyx.natural_language_processing.utils import get_tokenizer
-from onyx.server.query_and_chat.chat_utils import mime_type_to_chat_file_type
 from onyx.tools.models import ToolCallInfo
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string
 
 logger = setup_logger()
 
 
-def _extract_referenced_file_descriptors(
-    tool_calls: list[ToolCallInfo],
-    message_text: str,
-) -> list[FileDescriptor]:
-    """Extract FileDescriptors for code interpreter files referenced in the message text."""
-    descriptors: list[FileDescriptor] = []
-    for tool_call_info in tool_calls:
-        if not tool_call_info.generated_files:
-            continue
-        for gen_file in tool_call_info.generated_files:
-            file_id = (
-                gen_file.file_link.rsplit("/", 1)[-1] if gen_file.file_link else ""
-            )
-            if file_id and file_id in message_text:
-                mime_type, _ = mimetypes.guess_type(gen_file.filename)
-                descriptors.append(
-                    FileDescriptor(
-                        id=file_id,
-                        type=mime_type_to_chat_file_type(mime_type),
-                        name=gen_file.filename,
-                    )
-                )
-    return descriptors
-
-
 def _create_and_link_tool_calls(
     tool_calls: list[ToolCallInfo],
     assistant_message: ChatMessage,
@@ -202,13 +173,8 @@ def save_chat_turn(
         pre_answer_processing_time: Duration of processing before answer starts (in seconds)
     """
     # 1. Update ChatMessage with message content, reasoning tokens, and token count
-    sanitized_message_text = (
-        sanitize_string(message_text) if message_text else message_text
-    )
-    assistant_message.message = sanitized_message_text
-    assistant_message.reasoning_tokens = (
-        sanitize_string(reasoning_tokens) if reasoning_tokens else reasoning_tokens
-    )
+    assistant_message.message = message_text
+    assistant_message.reasoning_tokens = reasoning_tokens
     assistant_message.is_clarification = is_clarification
 
     # Use pre-answer processing time (captured when MESSAGE_START was emitted)
@@ -218,10 +184,8 @@ def save_chat_turn(
     # Calculate token count using default tokenizer, when storing, this should not use the LLM
     # specific one so we use a system default tokenizer here.
     default_tokenizer = get_tokenizer(None, None)
-    if sanitized_message_text:
-        assistant_message.token_count = len(
-            default_tokenizer.encode(sanitized_message_text)
-        )
+    if message_text:
+        assistant_message.token_count = len(default_tokenizer.encode(message_text))
     else:
         assistant_message.token_count = 0
 
@@ -333,16 +297,5 @@ def save_chat_turn(
         citation_number_to_search_doc_id if citation_number_to_search_doc_id else None
     )
 
-    # 8. Attach code interpreter generated files that the assistant actually
-    # referenced in its response, so they are available via load_all_chat_files
-    # on subsequent turns. Files not mentioned are intermediate artifacts.
-    if sanitized_message_text:
-        referenced = _extract_referenced_file_descriptors(
-            tool_calls, sanitized_message_text
-        )
-        if referenced:
-            existing_files = assistant_message.files or []
-            assistant_message.files = existing_files + referenced
-
     # Finally save the messages, tool calls, and docs
     db_session.commit()

backend/onyx/chat/stop_signal_checker.py

@@ -1,58 +1,65 @@
 from uuid import UUID
 
-from onyx.cache.interface import CacheBackend
+from redis.client import Redis
 
+# Redis key prefixes for chat session stop signals
 PREFIX = "chatsessionstop"
 FENCE_PREFIX = f"{PREFIX}_fence"
-FENCE_TTL = 10 * 60  # 10 minutes
+FENCE_TTL = 10 * 60  # 10 minutes - defensive TTL to prevent memory leaks
 
 
 def _get_fence_key(chat_session_id: UUID) -> str:
-    """Generate the cache key for a chat session stop signal fence.
+    """
+    Generate the Redis key for a chat session stop signal fence.
 
     Args:
         chat_session_id: The UUID of the chat session
 
     Returns:
-        The fence key string. Tenant isolation is handled automatically
-        by the cache backend (Redis key-prefixing or Postgres schema routing).
+        The fence key string (tenant_id is automatically added by the Redis client)
     """
     return f"{FENCE_PREFIX}_{chat_session_id}"
 
 
-def set_fence(chat_session_id: UUID, cache: CacheBackend, value: bool) -> None:
-    """Set or clear the stop signal fence for a chat session.
+def set_fence(chat_session_id: UUID, redis_client: Redis, value: bool) -> None:
+    """
+    Set or clear the stop signal fence for a chat session.
 
     Args:
         chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use (tenant-aware client that auto-prefixes keys)
         value: True to set the fence (stop signal), False to clear it
     """
     fence_key = _get_fence_key(chat_session_id)
     if not value:
-        cache.delete(fence_key)
+        redis_client.delete(fence_key)
         return
-    cache.set(fence_key, 0, ex=FENCE_TTL)
+
+    redis_client.set(fence_key, 0, ex=FENCE_TTL)
 
 
-def is_connected(chat_session_id: UUID, cache: CacheBackend) -> bool:
-    """Check if the chat session should continue (not stopped).
+def is_connected(chat_session_id: UUID, redis_client: Redis) -> bool:
+    """
+    Check if the chat session should continue (not stopped).
 
     Args:
         chat_session_id: The UUID of the chat session to check
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use for checking the stop signal (tenant-aware client that auto-prefixes keys)
 
     Returns:
         True if the session should continue, False if it should stop
     """
-    return not cache.exists(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    return not bool(redis_client.exists(fence_key))
 
 
-def reset_cancel_status(chat_session_id: UUID, cache: CacheBackend) -> None:
-    """Clear the stop signal for a chat session.
+def reset_cancel_status(chat_session_id: UUID, redis_client: Redis) -> None:
+    """
+    Clear the stop signal for a chat session.
 
     Args:
         chat_session_id: The UUID of the chat session
-        cache: Tenant-aware cache backend
+        redis_client: Redis client to use (tenant-aware client that auto-prefixes keys)
     """
-    cache.delete(_get_fence_key(chat_session_id))
+    fence_key = _get_fence_key(chat_session_id)
+    redis_client.delete(fence_key)

backend/onyx/chat/tool_call_args_streaming.py

@@ -1,77 +0,0 @@
-from collections.abc import Generator
-from collections.abc import Mapping
-from typing import Any
-from typing import Type
-
-from onyx.llm.model_response import ChatCompletionDeltaToolCall
-from onyx.server.query_and_chat.placement import Placement
-from onyx.server.query_and_chat.streaming_models import Packet
-from onyx.server.query_and_chat.streaming_models import ToolCallArgumentDelta
-from onyx.tools.built_in_tools import TOOL_NAME_TO_CLASS
-from onyx.tools.interface import Tool
-from onyx.utils.jsonriver import Parser
-
-
-def _get_tool_class(
-    tool_calls_in_progress: Mapping[int, Mapping[str, Any]],
-    tool_call_delta: ChatCompletionDeltaToolCall,
-) -> Type[Tool] | None:
-    """Look up the Tool subclass for a streaming tool call delta."""
-    tool_name = tool_calls_in_progress.get(tool_call_delta.index, {}).get("name")
-    if not tool_name:
-        return None
-    return TOOL_NAME_TO_CLASS.get(tool_name)
-
-
-def maybe_emit_argument_delta(
-    tool_calls_in_progress: Mapping[int, Mapping[str, Any]],
-    tool_call_delta: ChatCompletionDeltaToolCall,
-    placement: Placement,
-    parsers: dict[int, Parser],
-) -> Generator[Packet, None, None]:
-    """Emit decoded tool-call argument deltas to the frontend.
-
-    Uses a ``jsonriver.Parser`` per tool-call index to incrementally parse
-    the JSON argument string and extract only the newly-appended content
-    for each string-valued argument.
-
-    NOTE: Non-string arguments (numbers, booleans, null, arrays, objects)
-    are skipped — they are available in the final tool-call kickoff packet.
-
-    ``parsers`` is a mutable dict keyed by tool-call index. A new
-    ``Parser`` is created automatically for each new index.
-    """
-    tool_cls = _get_tool_class(tool_calls_in_progress, tool_call_delta)
-    if not tool_cls or not tool_cls.should_emit_argument_deltas():
-        return
-
-    fn = tool_call_delta.function
-    delta_fragment = fn.arguments if fn else None
-    if not delta_fragment:
-        return
-
-    idx = tool_call_delta.index
-    if idx not in parsers:
-        parsers[idx] = Parser()
-    parser = parsers[idx]
-
-    deltas = parser.feed(delta_fragment)
-
-    argument_deltas: dict[str, str] = {}
-    for delta in deltas:
-        if isinstance(delta, dict):
-            for key, value in delta.items():
-                if isinstance(value, str):
-                    argument_deltas[key] = argument_deltas.get(key, "") + value
-
-    if not argument_deltas:
-        return
-
-    tc_data = tool_calls_in_progress[tool_call_delta.index]
-    yield Packet(
-        placement=placement,
-        obj=ToolCallArgumentDelta(
-            tool_type=tc_data.get("name", ""),
-            argument_deltas=argument_deltas,
-        ),
-    )

backend/onyx/configs/app_configs.py

@@ -68,10 +68,6 @@ FILE_TOKEN_COUNT_THRESHOLD = int(
     os.environ.get("FILE_TOKEN_COUNT_THRESHOLD", str(_DEFAULT_FILE_TOKEN_LIMIT))
 )
 
-# Maximum upload size for a single user file (chat/projects) in MB.
-USER_FILE_MAX_UPLOAD_SIZE_MB = int(os.environ.get("USER_FILE_MAX_UPLOAD_SIZE_MB") or 50)
-USER_FILE_MAX_UPLOAD_SIZE_BYTES = USER_FILE_MAX_UPLOAD_SIZE_MB * 1024 * 1024
-
 # If set to true, will show extra/uncommon connectors in the "Other" category
 SHOW_EXTRA_CONNECTORS = os.environ.get("SHOW_EXTRA_CONNECTORS", "").lower() == "true"
 
@@ -96,12 +92,19 @@ WEB_DOMAIN = os.environ.get("WEB_DOMAIN") or "http://localhost:3000"
 #####
 # Auth Configs
 #####
-# Silently default to basic - warnings/errors logged in verify_auth_setting()
-# which only runs on app startup, not during migrations/scripts
-_auth_type_str = (os.environ.get("AUTH_TYPE") or "").lower()
-if _auth_type_str in [auth_type.value for auth_type in AuthType]:
+# Upgrades users from disabled auth to basic auth and shows warning.
+_auth_type_str = (os.environ.get("AUTH_TYPE") or "basic").lower()
+if _auth_type_str == "disabled":
+    logger.warning(
+        "AUTH_TYPE='disabled' is no longer supported. "
+        "Defaulting to 'basic'. Please update your configuration. "
+        "Your existing data will be migrated automatically."
+    )
+    _auth_type_str = AuthType.BASIC.value
+try:
     AUTH_TYPE = AuthType(_auth_type_str)
-else:
+except ValueError:
+    logger.error(f"Invalid AUTH_TYPE: {_auth_type_str}. Defaulting to 'basic'.")
     AUTH_TYPE = AuthType.BASIC
 
 PASSWORD_MIN_LENGTH = int(os.getenv("PASSWORD_MIN_LENGTH", 8))
@@ -196,10 +199,6 @@ if _OIDC_SCOPE_OVERRIDE:
     except Exception:
         pass
 
-# Enables PKCE for OIDC login flow. Disabled by default to preserve
-# backwards compatibility for existing OIDC deployments.
-OIDC_PKCE_ENABLED = os.environ.get("OIDC_PKCE_ENABLED", "").lower() == "true"
-
 # Applicable for SAML Auth
 SAML_CONF_DIR = os.environ.get("SAML_CONF_DIR") or "/app/onyx/configs/saml_config"
 
@@ -208,12 +207,6 @@ JWT_PUBLIC_KEY_URL: str | None = os.getenv("JWT_PUBLIC_KEY_URL", None)
 
 USER_AUTH_SECRET = os.environ.get("USER_AUTH_SECRET", "")
 
-if AUTH_TYPE == AuthType.BASIC and not USER_AUTH_SECRET:
-    logger.warning(
-        "USER_AUTH_SECRET is not set. This is required for secure password reset "
-        "and email verification tokens. Please set USER_AUTH_SECRET in production."
-    )
-
 # Duration (in seconds) for which the FastAPI Users JWT token remains valid in the user's browser.
 # By default, this is set to match the Redis expiry time for consistency.
 AUTH_COOKIE_EXPIRE_TIME_SECONDS = int(
@@ -295,9 +288,8 @@ OPENSEARCH_TEXT_ANALYZER = os.environ.get("OPENSEARCH_TEXT_ANALYZER") or "englis
 # environments we always want to be dual indexing into both OpenSearch and Vespa
 # to stress test the new codepaths. Only enable this if there is some instance
 # of OpenSearch running for the relevant Onyx instance.
-# NOTE: Now enabled on by default, unless the env indicates otherwise.
 ENABLE_OPENSEARCH_INDEXING_FOR_ONYX = (
-    os.environ.get("ENABLE_OPENSEARCH_INDEXING_FOR_ONYX", "true").lower() == "true"
+    os.environ.get("ENABLE_OPENSEARCH_INDEXING_FOR_ONYX", "").lower() == "true"
 )
 # NOTE: This effectively does nothing anymore, admins can now toggle whether
 # retrieval is through OpenSearch. This value is only used as a final fallback
@@ -315,12 +307,6 @@ VERIFY_CREATE_OPENSEARCH_INDEX_ON_INIT_MT = (
     os.environ.get("VERIFY_CREATE_OPENSEARCH_INDEX_ON_INIT_MT", "true").lower()
     == "true"
 )
-OPENSEARCH_MIGRATION_GET_VESPA_CHUNKS_PAGE_SIZE = int(
-    os.environ.get("OPENSEARCH_MIGRATION_GET_VESPA_CHUNKS_PAGE_SIZE") or 500
-)
-OPENSEARCH_OVERRIDE_DEFAULT_NUM_HYBRID_SEARCH_CANDIDATES = int(
-    os.environ.get("OPENSEARCH_DEFAULT_NUM_HYBRID_SEARCH_CANDIDATES") or 0
-)
 
 VESPA_HOST = os.environ.get("VESPA_HOST") or "localhost"
 # NOTE: this is used if and only if the vespa config server is accessible via a
@@ -509,7 +495,14 @@ CELERY_WORKER_PRIMARY_POOL_OVERFLOW = int(
     os.environ.get("CELERY_WORKER_PRIMARY_POOL_OVERFLOW") or 4
 )
 
-# Individual worker concurrency settings
+# Consolidated background worker (light, docprocessing, docfetching, heavy, monitoring, user_file_processing)
+# separate workers' defaults: light=24, docprocessing=6, docfetching=1, heavy=4, kg=2, monitoring=1, user_file=2
+# Total would be 40, but we use a more conservative default of 20 for the consolidated worker
+CELERY_WORKER_BACKGROUND_CONCURRENCY = int(
+    os.environ.get("CELERY_WORKER_BACKGROUND_CONCURRENCY") or 20
+)
+
+# Individual worker concurrency settings (used when USE_LIGHTWEIGHT_BACKGROUND_WORKER is False or on Kuberenetes deployments)
 CELERY_WORKER_HEAVY_CONCURRENCY = int(
     os.environ.get("CELERY_WORKER_HEAVY_CONCURRENCY") or 4
 )
@@ -826,9 +819,7 @@ RERANK_COUNT = int(os.environ.get("RERANK_COUNT") or 1000)
 # Tool Configs
 #####
 # Code Interpreter Service Configuration
-CODE_INTERPRETER_BASE_URL = os.environ.get(
-    "CODE_INTERPRETER_BASE_URL", "http://localhost:8000"
-)
+CODE_INTERPRETER_BASE_URL = os.environ.get("CODE_INTERPRETER_BASE_URL")
 
 CODE_INTERPRETER_DEFAULT_TIMEOUT_MS = int(
     os.environ.get("CODE_INTERPRETER_DEFAULT_TIMEOUT_MS") or 60_000
@@ -909,9 +900,6 @@ CUSTOM_ANSWER_VALIDITY_CONDITIONS = json.loads(
 )
 
 VESPA_REQUEST_TIMEOUT = int(os.environ.get("VESPA_REQUEST_TIMEOUT") or "15")
-VESPA_MIGRATION_REQUEST_TIMEOUT_S = int(
-    os.environ.get("VESPA_MIGRATION_REQUEST_TIMEOUT_S") or "120"
-)
 
 SYSTEM_RECURSION_LIMIT = int(os.environ.get("SYSTEM_RECURSION_LIMIT") or "1000")
 

backend/onyx/configs/constants.py

@@ -84,6 +84,7 @@ POSTGRES_CELERY_WORKER_LIGHT_APP_NAME = "celery_worker_light"
 POSTGRES_CELERY_WORKER_DOCPROCESSING_APP_NAME = "celery_worker_docprocessing"
 POSTGRES_CELERY_WORKER_DOCFETCHING_APP_NAME = "celery_worker_docfetching"
 POSTGRES_CELERY_WORKER_INDEXING_CHILD_APP_NAME = "celery_worker_indexing_child"
+POSTGRES_CELERY_WORKER_BACKGROUND_APP_NAME = "celery_worker_background"
 POSTGRES_CELERY_WORKER_HEAVY_APP_NAME = "celery_worker_heavy"
 POSTGRES_CELERY_WORKER_MONITORING_APP_NAME = "celery_worker_monitoring"
 POSTGRES_CELERY_WORKER_USER_FILE_PROCESSING_APP_NAME = (

backend/onyx/connectors/confluence/connector.py

@@ -943,9 +943,6 @@ class ConfluenceConnector(
                         if include_permissions
                         else None
                     ),
-                    parent_hierarchy_raw_node_id=self._get_parent_hierarchy_raw_id(
-                        page
-                    ),
                 )
             )
 
@@ -995,7 +992,6 @@ class ConfluenceConnector(
                             if include_permissions
                             else None
                         ),
-                        parent_hierarchy_raw_node_id=page_id,
                     )
                 )
 

backend/onyx/connectors/discord/connector.py

@@ -1,5 +1,4 @@
 import asyncio
-from collections.abc import AsyncGenerator
 from collections.abc import AsyncIterable
 from collections.abc import Iterable
 from datetime import datetime
@@ -205,7 +204,7 @@ def _manage_async_retrieval(
 
     end_time: datetime | None = end
 
-    async def _async_fetch() -> AsyncGenerator[Document, None]:
+    async def _async_fetch() -> AsyncIterable[Document]:
         intents = Intents.default()
         intents.message_content = True
         async with Client(intents=intents) as discord_client:
@@ -228,23 +227,22 @@ def _manage_async_retrieval(
 
     def run_and_yield() -> Iterable[Document]:
         loop = asyncio.new_event_loop()
-        async_gen = _async_fetch()
         try:
+            # Get the async generator
+            async_gen = _async_fetch()
+            # Convert to AsyncIterator
+            async_iter = async_gen.__aiter__()
             while True:
                 try:
-                    doc = loop.run_until_complete(anext(async_gen))
+                    # Create a coroutine by calling anext with the async iterator
+                    next_coro = anext(async_iter)
+                    # Run the coroutine to get the next document
+                    doc = loop.run_until_complete(next_coro)
                     yield doc
                 except StopAsyncIteration:
                     break
         finally:
-            # Must close the async generator before the loop so the Discord
-            # client's `async with` block can await its shutdown coroutine.
-            # The nested try/finally ensures the loop always closes even if
-            # aclose() raises (same pattern as cursor.close() before conn.close()).
-            try:
-                loop.run_until_complete(async_gen.aclose())
-            finally:
-                loop.close()
+            loop.close()
 
     return run_and_yield()
 

backend/onyx/connectors/google_drive/connector.py

@@ -1722,7 +1722,6 @@ class GoogleDriveConnector(
                         primary_admin_email=self.primary_admin_email,
                         google_domain=self.google_domain,
                     ),
-                    retriever_email=file.user_email,
                 ):
                     slim_batch.append(doc)
 

backend/onyx/connectors/google_drive/doc_conversion.py

@@ -476,7 +476,6 @@ def _get_external_access_for_raw_gdrive_file(
     company_domain: str,
     retriever_drive_service: GoogleDriveService | None,
     admin_drive_service: GoogleDriveService,
-    fallback_user_email: str,
     add_prefix: bool = False,
 ) -> ExternalAccess:
     """
@@ -485,8 +484,6 @@ def _get_external_access_for_raw_gdrive_file(
     add_prefix: When True, prefix group IDs with source type (for indexing path).
                When False (default), leave unprefixed (for permission sync path
                where upsert_document_external_perms handles prefixing).
-    fallback_user_email: When permission info can't be retrieved (e.g. externally-owned
-               files), fall back to granting access to this user.
     """
     external_access_fn = cast(
         Callable[
@@ -495,7 +492,6 @@ def _get_external_access_for_raw_gdrive_file(
                 str,
                 GoogleDriveService | None,
                 GoogleDriveService,
-                str,
                 bool,
             ],
             ExternalAccess,
@@ -511,7 +507,6 @@ def _get_external_access_for_raw_gdrive_file(
         company_domain,
         retriever_drive_service,
         admin_drive_service,
-        fallback_user_email,
         add_prefix,
     )
 
@@ -677,7 +672,6 @@ def _convert_drive_item_to_document(
                     creds, user_email=permission_sync_context.primary_admin_email
                 ),
                 add_prefix=True,  # Indexing path - prefix here
-                fallback_user_email=retriever_email,
             )
             if permission_sync_context
             else None
@@ -759,7 +753,6 @@ def build_slim_document(
     # if not specified, we will not sync permissions
     # will also be a no-op if EE is not enabled
     permission_sync_context: PermissionSyncContext | None,
-    retriever_email: str,
 ) -> SlimDocument | None:
     if file.get("mimeType") in [DRIVE_FOLDER_TYPE, DRIVE_SHORTCUT_TYPE]:
         return None
@@ -781,7 +774,6 @@ def build_slim_document(
                 creds,
                 user_email=permission_sync_context.primary_admin_email,
             ),
-            fallback_user_email=retriever_email,
         )
         if permission_sync_context
         else None
@@ -789,5 +781,4 @@ def build_slim_document(
     return SlimDocument(
         id=onyx_document_id_from_drive_file(file),
         external_access=external_access,
-        parent_hierarchy_raw_node_id=(file.get("parents") or [None])[0],
     )

backend/onyx/connectors/google_utils/google_kv.py

@@ -44,7 +44,6 @@ from onyx.connectors.google_utils.shared_constants import (
 from onyx.db.credentials import update_credential_json
 from onyx.db.models import User
 from onyx.key_value_store.factory import get_kv_store
-from onyx.key_value_store.interface import unwrap_str
 from onyx.server.documents.models import CredentialBase
 from onyx.server.documents.models import GoogleAppCredentials
 from onyx.server.documents.models import GoogleServiceAccountKey
@@ -90,7 +89,7 @@ def _get_current_oauth_user(creds: OAuthCredentials, source: DocumentSource) ->
 
 
 def verify_csrf(credential_id: int, state: str) -> None:
-    csrf = unwrap_str(get_kv_store().load(KV_CRED_KEY.format(str(credential_id))))
+    csrf = get_kv_store().load(KV_CRED_KEY.format(str(credential_id)))
     if csrf != state:
         raise PermissionError(
             "State from Google Drive Connector callback does not match expected"
@@ -179,9 +178,7 @@ def get_auth_url(credential_id: int, source: DocumentSource) -> str:
     params = parse_qs(parsed_url.query)
 
     get_kv_store().store(
-        KV_CRED_KEY.format(credential_id),
-        {"value": params.get("state", [None])[0]},
-        encrypt=True,
+        KV_CRED_KEY.format(credential_id), params.get("state", [None])[0], encrypt=True
     )
     return str(auth_url)
 

backend/onyx/connectors/jira/connector.py

@@ -902,11 +902,6 @@ class JiraConnector(
                         external_access=self._get_project_permissions(
                             project_key, add_prefix=False
                         ),
-                        parent_hierarchy_raw_node_id=(
-                            self._get_parent_hierarchy_raw_node_id(issue, project_key)
-                            if project_key
-                            else None
-                        ),
                     )
                 )
                 current_offset += 1

backend/onyx/connectors/models.py

@@ -385,7 +385,6 @@ class IndexingDocument(Document):
 class SlimDocument(BaseModel):
     id: str
     external_access: ExternalAccess | None = None
-    parent_hierarchy_raw_node_id: str | None = None
 
 
 class HierarchyNode(BaseModel):

backend/onyx/connectors/sharepoint/connector.py

@@ -33,7 +33,6 @@ from office365.runtime.queries.client_query import ClientQuery  # type: ignore[i
 from office365.sharepoint.client_context import ClientContext  # type: ignore[import-untyped]
 from pydantic import BaseModel
 from pydantic import Field
-from requests.exceptions import HTTPError
 
 from onyx.configs.app_configs import INDEX_BATCH_SIZE
 from onyx.configs.app_configs import REQUEST_TIMEOUT_SECONDS
@@ -259,10 +258,6 @@ class SharepointConnectorCheckpoint(ConnectorCheckpoint):
     # Track yielded hierarchy nodes by their raw_node_id (URLs) to avoid duplicates
     seen_hierarchy_node_raw_ids: set[str] = Field(default_factory=set)
 
-    # Track yielded document IDs to avoid processing the same document twice.
-    # The Microsoft Graph delta API can return the same item on multiple pages.
-    seen_document_ids: set[str] = Field(default_factory=set)
-
 
 class SharepointAuthMethod(Enum):
     CLIENT_SECRET = "client_secret"
@@ -273,32 +268,6 @@ class SizeCapExceeded(Exception):
     """Exception raised when the size cap is exceeded."""
 
 
-def _log_and_raise_for_status(response: requests.Response) -> None:
-    """Log the response text and raise for status."""
-    try:
-        response.raise_for_status()
-    except Exception:
-        logger.error(f"HTTP request failed: {response.text}")
-        raise
-
-
-GRAPH_INVALID_REQUEST_CODE = "invalidRequest"
-
-
-def _is_graph_invalid_request(response: requests.Response) -> bool:
-    """Return True if the response body is the generic Graph API
-    ``{"error": {"code": "invalidRequest", "message": "Invalid request"}}``
-    shape. This particular error has no actionable inner error code and is
-    returned by the site-pages endpoint when a page has a corrupt canvas layout
-    (e.g. duplicate web-part IDs — see SharePoint/sp-dev-docs#8822)."""
-    try:
-        body = response.json()
-    except Exception:
-        return False
-    error = body.get("error", {})
-    return error.get("code") == GRAPH_INVALID_REQUEST_CODE
-
-
 def load_certificate_from_pfx(pfx_data: bytes, password: str) -> CertificateData | None:
     """Load certificate from .pfx file for MSAL authentication"""
     try:
@@ -375,7 +344,7 @@ def _probe_remote_size(url: str, timeout: int) -> int | None:
     """Determine remote size using HEAD or a range GET probe. Returns None if unknown."""
     try:
         head_resp = requests.head(url, timeout=timeout, allow_redirects=True)
-        _log_and_raise_for_status(head_resp)
+        head_resp.raise_for_status()
         cl = head_resp.headers.get("Content-Length")
         if cl and cl.isdigit():
             return int(cl)
@@ -390,7 +359,7 @@ def _probe_remote_size(url: str, timeout: int) -> int | None:
             timeout=timeout,
             stream=True,
         ) as range_resp:
-            _log_and_raise_for_status(range_resp)
+            range_resp.raise_for_status()
             cr = range_resp.headers.get("Content-Range")  # e.g., "bytes 0-0/12345"
             if cr and "/" in cr:
                 total = cr.split("/")[-1]
@@ -415,7 +384,7 @@ def _download_with_cap(url: str, timeout: int, cap: int) -> bytes:
     - Returns the full bytes if the content fits within `cap`.
     """
     with requests.get(url, stream=True, timeout=timeout) as resp:
-        _log_and_raise_for_status(resp)
+        resp.raise_for_status()
 
         # If the server provides Content-Length, prefer an early decision.
         cl_header = resp.headers.get("Content-Length")
@@ -459,7 +428,7 @@ def _download_via_graph_api(
     with requests.get(
         url, headers=headers, stream=True, timeout=REQUEST_TIMEOUT_SECONDS
     ) as resp:
-        _log_and_raise_for_status(resp)
+        resp.raise_for_status()
         buf = io.BytesIO()
         for chunk in resp.iter_content(64 * 1024):
             if not chunk:
@@ -803,7 +772,6 @@ def _convert_driveitem_to_slim_document(
     drive_name: str,
     ctx: ClientContext,
     graph_client: GraphClient,
-    parent_hierarchy_raw_node_id: str | None = None,
 ) -> SlimDocument:
     if driveitem.id is None:
         raise ValueError("DriveItem ID is required")
@@ -819,15 +787,11 @@ def _convert_driveitem_to_slim_document(
     return SlimDocument(
         id=driveitem.id,
         external_access=external_access,
-        parent_hierarchy_raw_node_id=parent_hierarchy_raw_node_id,
     )
 
 
 def _convert_sitepage_to_slim_document(
-    site_page: dict[str, Any],
-    ctx: ClientContext | None,
-    graph_client: GraphClient,
-    parent_hierarchy_raw_node_id: str | None = None,
+    site_page: dict[str, Any], ctx: ClientContext | None, graph_client: GraphClient
 ) -> SlimDocument:
     """Convert a SharePoint site page to a SlimDocument object."""
     if site_page.get("id") is None:
@@ -844,7 +808,6 @@ def _convert_sitepage_to_slim_document(
     return SlimDocument(
         id=id,
         external_access=external_access,
-        parent_hierarchy_raw_node_id=parent_hierarchy_raw_node_id,
     )
 
 
@@ -1269,135 +1232,26 @@ class SharepointConnector(
         site.execute_query()
         site_id = site.id
 
-        site_pages_base = (
-            f"{self.graph_api_base}/sites/{site_id}/pages/microsoft.graph.sitePage"
+        page_url: str | None = (
+            f"{self.graph_api_base}/sites/{site_id}" f"/pages/microsoft.graph.sitePage"
         )
-        page_url: str | None = site_pages_base
         params: dict[str, str] | None = {"$expand": "canvasLayout"}
         total_yielded = 0
-        yielded_ids: set[str] = set()
 
         while page_url:
-            try:
-                data = self._graph_api_get_json(page_url, params)
-            except HTTPError as e:
-                if e.response is not None and e.response.status_code == 404:
-                    logger.warning(f"Site page not found: {page_url}")
-                    break
-                if (
-                    e.response is not None
-                    and e.response.status_code == 400
-                    and _is_graph_invalid_request(e.response)
-                ):
-                    logger.warning(
-                        f"$expand=canvasLayout on the LIST endpoint returned 400 "
-                        f"for site {site_descriptor.url}. Falling back to "
-                        f"per-page expansion."
-                    )
-                    yield from self._fetch_site_pages_individually(
-                        site_pages_base, start, end, skip_ids=yielded_ids
-                    )
-                    return
-                raise
-
+            data = self._graph_api_get_json(page_url, params)
             params = None  # nextLink already embeds query params
 
             for page in data.get("value", []):
                 if not _site_page_in_time_window(page, start, end):
                     continue
                 total_yielded += 1
-                page_id = page.get("id")
-                if page_id:
-                    yielded_ids.add(page_id)
                 yield page
 
             page_url = data.get("@odata.nextLink")
 
         logger.debug(f"Yielded {total_yielded} site pages for {site_descriptor.url}")
 
-    def _fetch_site_pages_individually(
-        self,
-        site_pages_base: str,
-        start: datetime | None = None,
-        end: datetime | None = None,
-        skip_ids: set[str] | None = None,
-    ) -> Generator[dict[str, Any], None, None]:
-        """Fallback for _fetch_site_pages: list pages without $expand, then
-        expand canvasLayout on each page individually.
-
-        The Graph API's LIST endpoint can return 400 when $expand=canvasLayout
-        is used and *any* page in the site has a corrupt canvas layout (e.g.
-        duplicate web part IDs — see SharePoint/sp-dev-docs#8822). Since the
-        LIST expansion is all-or-nothing, a single bad page poisons the entire
-        response. This method works around it by fetching metadata first, then
-        expanding each page individually so only the broken page loses its
-        canvas content.
-
-        ``skip_ids`` contains page IDs already yielded by the caller before the
-        fallback was triggered, preventing duplicates.
-        """
-        page_url: str | None = site_pages_base
-        total_yielded = 0
-        _skip_ids = skip_ids or set()
-
-        while page_url:
-            try:
-                data = self._graph_api_get_json(page_url)
-            except HTTPError as e:
-                if e.response is not None and e.response.status_code == 404:
-                    break
-                raise
-
-            for page in data.get("value", []):
-                if not _site_page_in_time_window(page, start, end):
-                    continue
-
-                page_id = page.get("id")
-                if page_id and page_id in _skip_ids:
-                    continue
-
-                if not page_id:
-                    total_yielded += 1
-                    yield page
-                    continue
-
-                expanded = self._try_expand_single_page(site_pages_base, page_id, page)
-                total_yielded += 1
-                yield expanded
-
-            page_url = data.get("@odata.nextLink")
-
-        logger.debug(
-            f"Yielded {total_yielded} site pages (per-page expansion fallback)"
-        )
-
-    def _try_expand_single_page(
-        self,
-        site_pages_base: str,
-        page_id: str,
-        fallback_page: dict[str, Any],
-    ) -> dict[str, Any]:
-        """Try to GET a single page with $expand=canvasLayout. On 400, return
-        the metadata-only fallback so the page is still indexed (without canvas
-        content)."""
-        pages_collection = site_pages_base.removesuffix("/microsoft.graph.sitePage")
-        single_url = f"{pages_collection}/{page_id}/microsoft.graph.sitePage"
-        try:
-            return self._graph_api_get_json(single_url, {"$expand": "canvasLayout"})
-        except HTTPError as e:
-            if (
-                e.response is not None
-                and e.response.status_code == 400
-                and _is_graph_invalid_request(e.response)
-            ):
-                page_name = fallback_page.get("name", page_id)
-                logger.warning(
-                    f"$expand=canvasLayout failed for page '{page_name}' "
-                    f"({page_id}). Indexing metadata only."
-                )
-                return fallback_page
-            raise
-
     def _acquire_token(self) -> dict[str, Any]:
         """
         Acquire token via MSAL
@@ -1449,7 +1303,7 @@ class SharepointConnector(
                         access_token = self._get_graph_access_token()
                         headers = {"Authorization": f"Bearer {access_token}"}
                         continue
-                _log_and_raise_for_status(response)
+                response.raise_for_status()
                 return response.json()
             except (requests.ConnectionError, requests.Timeout):
                 if attempt < GRAPH_API_MAX_RETRIES:
@@ -1697,7 +1551,6 @@ class SharepointConnector(
         checkpoint.current_drive_id = None
         checkpoint.current_drive_web_url = None
         checkpoint.current_drive_delta_next_link = None
-        checkpoint.seen_document_ids.clear()
 
     def _fetch_slim_documents_from_sharepoint(self) -> GenerateSlimDocumentOutput:
         site_descriptors = self.site_descriptors or self.fetch_sites()
@@ -1741,22 +1594,12 @@ class SharepointConnector(
                             )
                         )
 
-                    parent_hierarchy_url: str | None = None
-                    if drive_web_url:
-                        parent_hierarchy_url = self._get_parent_hierarchy_url(
-                            site_url, drive_web_url, drive_name, driveitem
-                        )
-
                     try:
                         logger.debug(f"Processing: {driveitem.web_url}")
                         ctx = self._create_rest_client_context(site_descriptor.url)
                         doc_batch.append(
                             _convert_driveitem_to_slim_document(
-                                driveitem,
-                                drive_name,
-                                ctx,
-                                self.graph_client,
-                                parent_hierarchy_raw_node_id=parent_hierarchy_url,
+                                driveitem, drive_name, ctx, self.graph_client
                             )
                         )
                     except Exception as e:
@@ -1776,10 +1619,7 @@ class SharepointConnector(
                     ctx = self._create_rest_client_context(site_descriptor.url)
                     doc_batch.append(
                         _convert_sitepage_to_slim_document(
-                            site_page,
-                            ctx,
-                            self.graph_client,
-                            parent_hierarchy_raw_node_id=site_descriptor.url,
+                            site_page, ctx, self.graph_client
                         )
                     )
                     if len(doc_batch) >= SLIM_BATCH_SIZE:
@@ -2278,14 +2118,6 @@ class SharepointConnector(
             item_count = 0
             for driveitem in driveitems:
                 item_count += 1
-
-                if driveitem.id and driveitem.id in checkpoint.seen_document_ids:
-                    logger.debug(
-                        f"Skipping duplicate document {driveitem.id} "
-                        f"({driveitem.name})"
-                    )
-                    continue
-
                 driveitem_extension = get_file_ext(driveitem.name)
                 if driveitem_extension not in OnyxFileExtensions.ALL_ALLOWED_EXTENSIONS:
                     logger.warning(
@@ -2338,13 +2170,11 @@ class SharepointConnector(
 
                     if isinstance(doc_or_failure, Document):
                         if doc_or_failure.sections:
-                            checkpoint.seen_document_ids.add(doc_or_failure.id)
                             yield doc_or_failure
                         elif should_yield_if_empty:
                             doc_or_failure.sections = [
                                 TextSection(link=driveitem.web_url, text="")
                             ]
-                            checkpoint.seen_document_ids.add(doc_or_failure.id)
                             yield doc_or_failure
                         else:
                             logger.warning(

backend/onyx/connectors/slack/connector.py

@@ -565,7 +565,6 @@ def _get_all_doc_ids(
                             channel_id=channel_id, thread_ts=message["ts"]
                         ),
                         external_access=external_access,
-                        parent_hierarchy_raw_node_id=channel_id,
                     )
                 )
 

backend/onyx/db/chat.py

@@ -38,7 +38,6 @@ from onyx.llm.override_models import LLMOverride
 from onyx.llm.override_models import PromptOverride
 from onyx.server.query_and_chat.models import ChatMessageDetail
 from onyx.utils.logger import setup_logger
-from onyx.utils.postgres_sanitization import sanitize_string
 
 
 logger = setup_logger()
@@ -676,43 +675,58 @@ def set_as_latest_chat_message(
     db_session.commit()
 
 
+def _sanitize_for_postgres(value: str) -> str:
+    """Remove NUL (0x00) characters from strings as PostgreSQL doesn't allow them."""
+    sanitized = value.replace("\x00", "")
+    if value and not sanitized:
+        logger.warning("Sanitization removed all characters from string")
+    return sanitized
+
+
+def _sanitize_list_for_postgres(values: list[str]) -> list[str]:
+    """Remove NUL (0x00) characters from all strings in a list."""
+    return [_sanitize_for_postgres(v) for v in values]
+
+
 def create_db_search_doc(
     server_search_doc: ServerSearchDoc,
     db_session: Session,
     commit: bool = True,
 ) -> DBSearchDoc:
+    # Sanitize string fields to remove NUL characters (PostgreSQL doesn't allow them)
     db_search_doc = DBSearchDoc(
-        document_id=sanitize_string(server_search_doc.document_id),
+        document_id=_sanitize_for_postgres(server_search_doc.document_id),
         chunk_ind=server_search_doc.chunk_ind,
-        semantic_id=sanitize_string(server_search_doc.semantic_identifier),
+        semantic_id=_sanitize_for_postgres(server_search_doc.semantic_identifier),
         link=(
-            sanitize_string(server_search_doc.link)
+            _sanitize_for_postgres(server_search_doc.link)
             if server_search_doc.link is not None
             else None
         ),
-        blurb=sanitize_string(server_search_doc.blurb),
+        blurb=_sanitize_for_postgres(server_search_doc.blurb),
         source_type=server_search_doc.source_type,
         boost=server_search_doc.boost,
         hidden=server_search_doc.hidden,
         doc_metadata=server_search_doc.metadata,
         is_relevant=server_search_doc.is_relevant,
         relevance_explanation=(
-            sanitize_string(server_search_doc.relevance_explanation)
+            _sanitize_for_postgres(server_search_doc.relevance_explanation)
             if server_search_doc.relevance_explanation is not None
             else None
         ),
+        # For docs further down that aren't reranked, we can't use the retrieval score
         score=server_search_doc.score or 0.0,
-        match_highlights=[
-            sanitize_string(h) for h in server_search_doc.match_highlights
-        ],
+        match_highlights=_sanitize_list_for_postgres(
+            server_search_doc.match_highlights
+        ),
         updated_at=server_search_doc.updated_at,
         primary_owners=(
-            [sanitize_string(o) for o in server_search_doc.primary_owners]
+            _sanitize_list_for_postgres(server_search_doc.primary_owners)
             if server_search_doc.primary_owners is not None
             else None
         ),
         secondary_owners=(
-            [sanitize_string(o) for o in server_search_doc.secondary_owners]
+            _sanitize_list_for_postgres(server_search_doc.secondary_owners)
             if server_search_doc.secondary_owners is not None
             else None
         ),

backend/onyx/db/document_set.py

@@ -13,7 +13,6 @@ from sqlalchemy.orm import aliased
 from sqlalchemy.orm import selectinload
 from sqlalchemy.orm import Session
 
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.db.connector_credential_pair import get_cc_pair_groups_for_ids
 from onyx.db.connector_credential_pair import get_connector_credential_pairs
 from onyx.db.enums import AccessType
@@ -247,7 +246,6 @@ def insert_document_set(
             description=document_set_creation_request.description,
             user_id=user_id,
             is_public=document_set_creation_request.is_public,
-            is_up_to_date=DISABLE_VECTOR_DB,
             time_last_modified_by_user=func.now(),
         )
         db_session.add(new_document_set_row)
@@ -338,8 +336,7 @@ def update_document_set(
             )
 
         document_set_row.description = document_set_update_request.description
-        if not DISABLE_VECTOR_DB:
-            document_set_row.is_up_to_date = False
+        document_set_row.is_up_to_date = False
         document_set_row.is_public = document_set_update_request.is_public
         document_set_row.time_last_modified_by_user = func.now()
         versioned_private_doc_set_fn = fetch_versioned_implementation(

backend/onyx/db/enums.py

@@ -304,13 +304,3 @@ class LLMModelFlowType(str, PyEnum):
     CHAT = "chat"
     VISION = "vision"
     CONTEXTUAL_RAG = "contextual_rag"
-
-
-class HookPoint(str, PyEnum):
-    DOCUMENT_INGESTION = "document_ingestion"
-    QUERY_PROCESSING = "query_processing"
-
-
-class HookFailStrategy(str, PyEnum):
-    HARD = "hard"  # exception propagates, pipeline aborts
-    SOFT = "soft"  # log error, return original input, pipeline continues

backend/onyx/db/hierarchy.py

@@ -1,11 +1,6 @@
 """CRUD operations for HierarchyNode."""
 
-from collections import defaultdict
-
-from sqlalchemy import delete
 from sqlalchemy import select
-from sqlalchemy.dialects.postgresql import insert as pg_insert
-from sqlalchemy.engine import CursorResult
 from sqlalchemy.orm import Session
 
 from onyx.configs.constants import DocumentSource
@@ -13,7 +8,6 @@ from onyx.connectors.models import HierarchyNode as PydanticHierarchyNode
 from onyx.db.enums import HierarchyNodeType
 from onyx.db.models import Document
 from onyx.db.models import HierarchyNode
-from onyx.db.models import HierarchyNodeByConnectorCredentialPair
 from onyx.utils.logger import setup_logger
 from onyx.utils.variable_functionality import fetch_versioned_implementation
 
@@ -462,7 +456,7 @@ def get_all_hierarchy_nodes_for_source(
 def _get_accessible_hierarchy_nodes_for_source(
     db_session: Session,
     source: DocumentSource,
-    user_email: str,  # noqa: ARG001
+    user_email: str | None,  # noqa: ARG001
     external_group_ids: list[str],  # noqa: ARG001
 ) -> list[HierarchyNode]:
     """
@@ -489,7 +483,7 @@ def _get_accessible_hierarchy_nodes_for_source(
 def get_accessible_hierarchy_nodes_for_source(
     db_session: Session,
     source: DocumentSource,
-    user_email: str,
+    user_email: str | None,
     external_group_ids: list[str],
 ) -> list[HierarchyNode]:
     """
@@ -531,53 +525,6 @@ def get_document_parent_hierarchy_node_ids(
     return {doc_id: parent_id for doc_id, parent_id in results}
 
 
-def update_document_parent_hierarchy_nodes(
-    db_session: Session,
-    doc_parent_map: dict[str, int | None],
-    commit: bool = True,
-) -> int:
-    """Bulk-update Document.parent_hierarchy_node_id for multiple documents.
-
-    Only updates rows whose current value differs from the desired value to
-    avoid unnecessary writes.
-
-    Args:
-        db_session: SQLAlchemy session
-        doc_parent_map: Mapping of document_id → desired parent_hierarchy_node_id
-        commit: Whether to commit the transaction
-
-    Returns:
-        Number of documents actually updated
-    """
-    if not doc_parent_map:
-        return 0
-
-    doc_ids = list(doc_parent_map.keys())
-    existing = get_document_parent_hierarchy_node_ids(db_session, doc_ids)
-
-    by_parent: dict[int | None, list[str]] = defaultdict(list)
-    for doc_id, desired_parent_id in doc_parent_map.items():
-        current = existing.get(doc_id)
-        if current == desired_parent_id or doc_id not in existing:
-            continue
-        by_parent[desired_parent_id].append(doc_id)
-
-    updated = 0
-    for desired_parent_id, ids in by_parent.items():
-        db_session.query(Document).filter(Document.id.in_(ids)).update(
-            {Document.parent_hierarchy_node_id: desired_parent_id},
-            synchronize_session=False,
-        )
-        updated += len(ids)
-
-    if commit:
-        db_session.commit()
-    elif updated:
-        db_session.flush()
-
-    return updated
-
-
 def update_hierarchy_node_permissions(
     db_session: Session,
     raw_node_id: str,
@@ -624,154 +571,3 @@ def update_hierarchy_node_permissions(
         db_session.flush()
 
     return True
-
-
-def upsert_hierarchy_node_cc_pair_entries(
-    db_session: Session,
-    hierarchy_node_ids: list[int],
-    connector_id: int,
-    credential_id: int,
-    commit: bool = True,
-) -> None:
-    """Insert rows into HierarchyNodeByConnectorCredentialPair, ignoring conflicts.
-
-    This records that the given cc_pair "owns" these hierarchy nodes. Used by
-    indexing, pruning, and hierarchy-fetching paths.
-    """
-    if not hierarchy_node_ids:
-        return
-
-    _M = HierarchyNodeByConnectorCredentialPair
-    stmt = pg_insert(_M).values(
-        [
-            {
-                _M.hierarchy_node_id: node_id,
-                _M.connector_id: connector_id,
-                _M.credential_id: credential_id,
-            }
-            for node_id in hierarchy_node_ids
-        ]
-    )
-    stmt = stmt.on_conflict_do_nothing()
-    db_session.execute(stmt)
-
-    if commit:
-        db_session.commit()
-    else:
-        db_session.flush()
-
-
-def remove_stale_hierarchy_node_cc_pair_entries(
-    db_session: Session,
-    connector_id: int,
-    credential_id: int,
-    live_hierarchy_node_ids: set[int],
-    commit: bool = True,
-) -> int:
-    """Delete join-table rows for this cc_pair that are NOT in the live set.
-
-    If ``live_hierarchy_node_ids`` is empty ALL rows for the cc_pair are deleted
-    (i.e. the connector no longer has any hierarchy nodes). Callers that want a
-    no-op when there are no live nodes must guard before calling.
-
-    Returns the number of deleted rows.
-    """
-    stmt = delete(HierarchyNodeByConnectorCredentialPair).where(
-        HierarchyNodeByConnectorCredentialPair.connector_id == connector_id,
-        HierarchyNodeByConnectorCredentialPair.credential_id == credential_id,
-    )
-    if live_hierarchy_node_ids:
-        stmt = stmt.where(
-            HierarchyNodeByConnectorCredentialPair.hierarchy_node_id.notin_(
-                live_hierarchy_node_ids
-            )
-        )
-
-    result: CursorResult = db_session.execute(stmt)  # type: ignore[assignment]
-    deleted = result.rowcount
-
-    if commit:
-        db_session.commit()
-    elif deleted:
-        db_session.flush()
-
-    return deleted
-
-
-def delete_orphaned_hierarchy_nodes(
-    db_session: Session,
-    source: DocumentSource,
-    commit: bool = True,
-) -> list[str]:
-    """Delete hierarchy nodes for a source that have zero cc_pair associations.
-
-    SOURCE-type nodes are excluded (they are synthetic roots).
-
-    Returns the list of raw_node_ids that were deleted (for cache eviction).
-    """
-    # Find orphaned nodes: no rows in the join table
-    orphan_stmt = (
-        select(HierarchyNode.id, HierarchyNode.raw_node_id)
-        .outerjoin(
-            HierarchyNodeByConnectorCredentialPair,
-            HierarchyNode.id
-            == HierarchyNodeByConnectorCredentialPair.hierarchy_node_id,
-        )
-        .where(
-            HierarchyNode.source == source,
-            HierarchyNode.node_type != HierarchyNodeType.SOURCE,
-            HierarchyNodeByConnectorCredentialPair.hierarchy_node_id.is_(None),
-        )
-    )
-    orphans = db_session.execute(orphan_stmt).all()
-    if not orphans:
-        return []
-
-    orphan_ids = [row[0] for row in orphans]
-    deleted_raw_ids = [row[1] for row in orphans]
-
-    db_session.execute(delete(HierarchyNode).where(HierarchyNode.id.in_(orphan_ids)))
-
-    if commit:
-        db_session.commit()
-    else:
-        db_session.flush()
-
-    return deleted_raw_ids
-
-
-def reparent_orphaned_hierarchy_nodes(
-    db_session: Session,
-    source: DocumentSource,
-    commit: bool = True,
-) -> list[HierarchyNode]:
-    """Re-parent hierarchy nodes whose parent_id is NULL to the SOURCE node.
-
-    After pruning deletes stale nodes, their former children get parent_id=NULL
-    via the SET NULL cascade. This function points them back to the SOURCE root.
-
-    Returns the reparented HierarchyNode objects (with updated parent_id)
-    so callers can refresh downstream caches.
-    """
-    source_node = get_source_hierarchy_node(db_session, source)
-    if not source_node:
-        return []
-
-    stmt = select(HierarchyNode).where(
-        HierarchyNode.source == source,
-        HierarchyNode.parent_id.is_(None),
-        HierarchyNode.node_type != HierarchyNodeType.SOURCE,
-    )
-    orphans = list(db_session.execute(stmt).scalars().all())
-    if not orphans:
-        return []
-
-    for node in orphans:
-        node.parent_id = source_node.id
-
-    if commit:
-        db_session.commit()
-    else:
-        db_session.flush()
-
-    return orphans

backend/onyx/db/llm.py

@@ -25,12 +25,8 @@ from onyx.server.manage.embedding.models import CloudEmbeddingProvider
 from onyx.server.manage.embedding.models import CloudEmbeddingProviderCreationRequest
 from onyx.server.manage.llm.models import LLMProviderUpsertRequest
 from onyx.server.manage.llm.models import LLMProviderView
-from onyx.server.manage.llm.models import SyncModelEntry
-from onyx.utils.logger import setup_logger
 from shared_configs.enums import EmbeddingProvider
 
-logger = setup_logger()
-
 
 def update_group_llm_provider_relationships__no_commit(
     llm_provider_id: int,
@@ -271,35 +267,10 @@ def upsert_llm_provider(
         mc.name for mc in llm_provider_upsert_request.model_configurations
     }
 
-    # Build a lookup of requested visibility by model name
-    requested_visibility = {
-        mc.name: mc.is_visible
-        for mc in llm_provider_upsert_request.model_configurations
-    }
-
     # Delete removed models
     removed_ids = [
         mc.id for name, mc in existing_by_name.items() if name not in models_to_exist
     ]
-
-    default_model = fetch_default_llm_model(db_session)
-
-    # Prevent removing and hiding the default model
-    if default_model:
-        for name, mc in existing_by_name.items():
-            if mc.id == default_model.id:
-                if default_model.id in removed_ids:
-                    raise ValueError(
-                        f"Cannot remove the default model '{name}'. "
-                        "Please change the default model before removing."
-                    )
-                if not requested_visibility.get(name, True):
-                    raise ValueError(
-                        f"Cannot hide the default model '{name}'. "
-                        "Please change the default model before hiding."
-                    )
-                break
-
     if removed_ids:
         db_session.query(ModelConfiguration).filter(
             ModelConfiguration.id.in_(removed_ids)
@@ -370,9 +341,9 @@ def upsert_llm_provider(
 def sync_model_configurations(
     db_session: Session,
     provider_name: str,
-    models: list[SyncModelEntry],
+    models: list[dict],
 ) -> int:
-    """Sync model configurations for a dynamic provider (OpenRouter, Bedrock, Ollama, etc.).
+    """Sync model configurations for a dynamic provider (OpenRouter, Bedrock, Ollama).
 
     This inserts NEW models from the source API without overwriting existing ones.
     User preferences (is_visible, max_input_tokens) are preserved for existing models.
@@ -380,7 +351,7 @@ def sync_model_configurations(
     Args:
         db_session: Database session
         provider_name: Name of the LLM provider
-        models: List of SyncModelEntry objects describing the fetched models
+        models: List of model dicts with keys: name, display_name, max_input_tokens, supports_image_input
 
     Returns:
         Number of new models added
@@ -394,20 +365,21 @@ def sync_model_configurations(
 
     new_count = 0
     for model in models:
-        if model.name not in existing_names:
+        model_name = model["name"]
+        if model_name not in existing_names:
             # Insert new model with is_visible=False (user must explicitly enable)
             supported_flows = [LLMModelFlowType.CHAT]
-            if model.supports_image_input:
+            if model.get("supports_image_input", False):
                 supported_flows.append(LLMModelFlowType.VISION)
 
             insert_new_model_configuration__no_commit(
                 db_session=db_session,
                 llm_provider_id=provider.id,
-                model_name=model.name,
+                model_name=model_name,
                 supported_flows=supported_flows,
                 is_visible=False,
-                max_input_tokens=model.max_input_tokens,
-                display_name=model.display_name,
+                max_input_tokens=model.get("max_input_tokens"),
+                display_name=model.get("display_name"),
             )
             new_count += 1
 
@@ -560,9 +532,9 @@ def fetch_default_model(
 ) -> ModelConfiguration | None:
     model_config = db_session.scalar(
         select(ModelConfiguration)
-        .options(selectinload(ModelConfiguration.llm_provider))
         .join(LLMModelFlow)
         .where(
+            ModelConfiguration.is_visible == True,  # noqa: E712
             LLMModelFlow.llm_model_flow_type == flow_type,
             LLMModelFlow.is_default == True,  # noqa: E712
         )
@@ -838,29 +810,6 @@ def sync_auto_mode_models(
             )
             changes += 1
 
-    # Update the default if this provider currently holds the global CHAT default.
-    # We flush (but don't commit) so that _update_default_model can see the new
-    # model rows, then commit everything atomically to avoid a window where the
-    # old default is invisible but still pointed-to.
-    db_session.flush()
-
-    recommended_default = llm_recommendations.get_default_model(provider.provider)
-    if recommended_default:
-        current_default = fetch_default_llm_model(db_session)
-
-        if (
-            current_default
-            and current_default.llm_provider_id == provider.id
-            and current_default.name != recommended_default.name
-        ):
-            _update_default_model__no_commit(
-                db_session=db_session,
-                provider_id=provider.id,
-                model=recommended_default.name,
-                flow_type=LLMModelFlowType.CHAT,
-            )
-            changes += 1
-
     db_session.commit()
     return changes
 
@@ -992,7 +941,7 @@ def update_model_configuration__no_commit(
     db_session.flush()
 
 
-def _update_default_model__no_commit(
+def _update_default_model(
     db_session: Session,
     provider_id: int,
     model: str,
@@ -1030,14 +979,6 @@ def _update_default_model__no_commit(
     new_default.is_default = True
     model_config.is_visible = True
 
-
-def _update_default_model(
-    db_session: Session,
-    provider_id: int,
-    model: str,
-    flow_type: LLMModelFlowType,
-) -> None:
-    _update_default_model__no_commit(db_session, provider_id, model, flow_type)
     db_session.commit()
 
 

backend/onyx/db/models.py

@@ -25,7 +25,6 @@ from sqlalchemy import desc
 from sqlalchemy import Enum
 from sqlalchemy import Float
 from sqlalchemy import ForeignKey
-from sqlalchemy import ForeignKeyConstraint
 from sqlalchemy import func
 from sqlalchemy import Index
 from sqlalchemy import Integer
@@ -37,11 +36,9 @@ from sqlalchemy import Text
 from sqlalchemy import text
 from sqlalchemy import UniqueConstraint
 from sqlalchemy.dialects import postgresql
-from sqlalchemy import event
 from sqlalchemy.engine.interfaces import Dialect
 from sqlalchemy.orm import DeclarativeBase
 from sqlalchemy.orm import Mapped
-from sqlalchemy.orm import Mapper
 from sqlalchemy.orm import mapped_column
 from sqlalchemy.orm import relationship
 from sqlalchemy.types import LargeBinary
@@ -64,8 +61,6 @@ from onyx.db.enums import (
     BuildSessionStatus,
     EmbeddingPrecision,
     HierarchyNodeType,
-    HookFailStrategy,
-    HookPoint,
     IndexingMode,
     OpenSearchDocumentMigrationStatus,
     OpenSearchTenantMigrationStatus,
@@ -122,52 +117,10 @@ class Base(DeclarativeBase):
     __abstract__ = True
 
 
-class _EncryptedBase(TypeDecorator):
-    """Base for encrypted column types that wrap values in SensitiveValue."""
-
+class EncryptedString(TypeDecorator):
     impl = LargeBinary
+    # This type's behavior is fully deterministic and doesn't depend on any external factors.
     cache_ok = True
-    _is_json: bool = False
-
-    def wrap_raw(self, value: Any) -> SensitiveValue:
-        """Encrypt a raw value and wrap it in SensitiveValue.
-
-        Called by the attribute set event so the Python-side type is always
-        SensitiveValue, regardless of whether the value was loaded from the DB
-        or assigned in application code.
-        """
-        if self._is_json:
-            if not isinstance(value, dict):
-                raise TypeError(
-                    f"EncryptedJson column expected dict, got {type(value).__name__}"
-                )
-            raw_str = json.dumps(value)
-        else:
-            if not isinstance(value, str):
-                raise TypeError(
-                    f"EncryptedString column expected str, got {type(value).__name__}"
-                )
-            raw_str = value
-        return SensitiveValue(
-            encrypted_bytes=encrypt_string_to_bytes(raw_str),
-            decrypt_fn=decrypt_bytes_to_string,
-            is_json=self._is_json,
-        )
-
-    def compare_values(self, x: Any, y: Any) -> bool:
-        if x is None or y is None:
-            return x == y
-        if isinstance(x, SensitiveValue):
-            x = x.get_value(apply_mask=False)
-        if isinstance(y, SensitiveValue):
-            y = y.get_value(apply_mask=False)
-        return x == y
-
-
-class EncryptedString(_EncryptedBase):
-    # Must redeclare cache_ok in this child class since we explicitly redeclare _is_json
-    cache_ok = True
-    _is_json: bool = False
 
     def process_bind_param(
         self, value: str | SensitiveValue[str] | None, dialect: Dialect  # noqa: ARG002
@@ -191,10 +144,20 @@ class EncryptedString(_EncryptedBase):
             )
         return None
 
+    def compare_values(self, x: Any, y: Any) -> bool:
+        if x is None or y is None:
+            return x == y
+        if isinstance(x, SensitiveValue):
+            x = x.get_value(apply_mask=False)
+        if isinstance(y, SensitiveValue):
+            y = y.get_value(apply_mask=False)
+        return x == y
 
-class EncryptedJson(_EncryptedBase):
+
+class EncryptedJson(TypeDecorator):
+    impl = LargeBinary
+    # This type's behavior is fully deterministic and doesn't depend on any external factors.
     cache_ok = True
-    _is_json: bool = True
 
     def process_bind_param(
         self,
@@ -202,7 +165,9 @@ class EncryptedJson(_EncryptedBase):
         dialect: Dialect,  # noqa: ARG002
     ) -> bytes | None:
         if value is not None:
+            # Handle both raw dicts and SensitiveValue wrappers
             if isinstance(value, SensitiveValue):
+                # Get raw value for storage
                 value = value.get_value(apply_mask=False)
             json_str = json.dumps(value)
             return encrypt_string_to_bytes(json_str)
@@ -219,40 +184,14 @@ class EncryptedJson(_EncryptedBase):
             )
         return None
 
-
-_REGISTERED_ATTRS: set[str] = set()
-
-
-@event.listens_for(Mapper, "mapper_configured")
-def _register_sensitive_value_set_events(
-    mapper: Mapper,
-    class_: type,
-) -> None:
-    """Auto-wrap raw values in SensitiveValue when assigned to encrypted columns."""
-    for prop in mapper.column_attrs:
-        for col in prop.columns:
-            if isinstance(col.type, _EncryptedBase):
-                col_type = col.type
-                attr = getattr(class_, prop.key)
-
-                # Guard against double-registration (e.g. if mapper is
-                # re-configured in test setups)
-                attr_key = f"{class_.__qualname__}.{prop.key}"
-                if attr_key in _REGISTERED_ATTRS:
-                    continue
-                _REGISTERED_ATTRS.add(attr_key)
-
-                @event.listens_for(attr, "set", retval=True)
-                def _wrap_value(
-                    target: Any,  # noqa: ARG001
-                    value: Any,
-                    oldvalue: Any,  # noqa: ARG001
-                    initiator: Any,  # noqa: ARG001
-                    _col_type: _EncryptedBase = col_type,
-                ) -> Any:
-                    if value is not None and not isinstance(value, SensitiveValue):
-                        return _col_type.wrap_raw(value)
-                    return value
+    def compare_values(self, x: Any, y: Any) -> bool:
+        if x is None or y is None:
+            return x == y
+        if isinstance(x, SensitiveValue):
+            x = x.get_value(apply_mask=False)
+        if isinstance(y, SensitiveValue):
+            y = y.get_value(apply_mask=False)
+        return x == y
 
 
 class NullFilteredString(TypeDecorator):
@@ -341,25 +280,10 @@ class User(SQLAlchemyBaseUserTableUUID, Base):
         TIMESTAMPAware(timezone=True), nullable=True
     )
 
-    created_at: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True), server_default=func.now(), nullable=False
-    )
-    updated_at: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True),
-        server_default=func.now(),
-        onupdate=func.now(),
-        nullable=False,
-    )
-
     default_model: Mapped[str] = mapped_column(Text, nullable=True)
     # organized in typical structured fashion
     # formatted as `displayName__provider__modelName`
 
-    # Voice preferences
-    voice_auto_send: Mapped[bool] = mapped_column(Boolean, default=False)
-    voice_auto_playback: Mapped[bool] = mapped_column(Boolean, default=False)
-    voice_playback_speed: Mapped[float] = mapped_column(Float, default=1.0)
-
     # relationships
     credentials: Mapped[list["Credential"]] = relationship(
         "Credential", back_populates="user"
@@ -2446,38 +2370,6 @@ class SyncRecord(Base):
     )
 
 
-class HierarchyNodeByConnectorCredentialPair(Base):
-    """Tracks which cc_pairs reference each hierarchy node.
-
-    During pruning, stale entries are removed for the current cc_pair.
-    Hierarchy nodes with zero remaining entries are then deleted.
-    """
-
-    __tablename__ = "hierarchy_node_by_connector_credential_pair"
-
-    hierarchy_node_id: Mapped[int] = mapped_column(
-        ForeignKey("hierarchy_node.id", ondelete="CASCADE"), primary_key=True
-    )
-    connector_id: Mapped[int] = mapped_column(primary_key=True)
-    credential_id: Mapped[int] = mapped_column(primary_key=True)
-
-    __table_args__ = (
-        ForeignKeyConstraint(
-            ["connector_id", "credential_id"],
-            [
-                "connector_credential_pair.connector_id",
-                "connector_credential_pair.credential_id",
-            ],
-            ondelete="CASCADE",
-        ),
-        Index(
-            "ix_hierarchy_node_cc_pair_connector_credential",
-            "connector_id",
-            "credential_id",
-        ),
-    )
-
-
 class DocumentByConnectorCredentialPair(Base):
     """Represents an indexing of a document by a specific connector / credential pair"""
 
@@ -2930,17 +2822,8 @@ class LLMProvider(Base):
         postgresql.JSONB(), nullable=True
     )
 
-    # Deprecated: use LLMModelFlow with CHAT flow type instead
-    default_model_name: Mapped[str | None] = mapped_column(String, nullable=True)
-
     deployment_name: Mapped[str | None] = mapped_column(String, nullable=True)
 
-    # Deprecated: use LLMModelFlow.is_default with CHAT flow type instead
-    is_default_provider: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
-    # Deprecated: use LLMModelFlow.is_default with VISION flow type instead
-    is_default_vision_provider: Mapped[bool | None] = mapped_column(Boolean)
-    # Deprecated: use LLMModelFlow with VISION flow type instead
-    default_vision_model: Mapped[str | None] = mapped_column(String, nullable=True)
     # EE only
     is_public: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
     # Auto mode: models, visibility, and defaults are managed by GitHub config
@@ -3072,65 +2955,6 @@ class ImageGenerationConfig(Base):
     )
 
 
-class VoiceProvider(Base):
-    """Configuration for voice services (STT and TTS)."""
-
-    __tablename__ = "voice_provider"
-
-    id: Mapped[int] = mapped_column(Integer, primary_key=True)
-    name: Mapped[str] = mapped_column(String, unique=True)
-    provider_type: Mapped[str] = mapped_column(
-        String
-    )  # "openai", "azure", "elevenlabs"
-    api_key: Mapped[SensitiveValue[str] | None] = mapped_column(
-        EncryptedString(), nullable=True
-    )
-    api_base: Mapped[str | None] = mapped_column(String, nullable=True)
-    custom_config: Mapped[dict[str, Any] | None] = mapped_column(
-        postgresql.JSONB(), nullable=True
-    )
-
-    # Model/voice configuration
-    stt_model: Mapped[str | None] = mapped_column(
-        String, nullable=True
-    )  # e.g., "whisper-1"
-    tts_model: Mapped[str | None] = mapped_column(
-        String, nullable=True
-    )  # e.g., "tts-1", "tts-1-hd"
-    default_voice: Mapped[str | None] = mapped_column(
-        String, nullable=True
-    )  # e.g., "alloy", "echo"
-
-    # STT and TTS can use different providers - only one provider per type
-    is_default_stt: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
-    is_default_tts: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
-
-    deleted: Mapped[bool] = mapped_column(Boolean, default=False)
-
-    time_created: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True), server_default=func.now()
-    )
-    time_updated: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
-    )
-
-    # Enforce only one default STT provider and one default TTS provider at DB level
-    __table_args__ = (
-        Index(
-            "ix_voice_provider_one_default_stt",
-            "is_default_stt",
-            unique=True,
-            postgresql_where=(is_default_stt == True),  # noqa: E712
-        ),
-        Index(
-            "ix_voice_provider_one_default_tts",
-            "is_default_tts",
-            unique=True,
-            postgresql_where=(is_default_tts == True),  # noqa: E712
-        ),
-    )
-
-
 class CloudEmbeddingProvider(Base):
     __tablename__ = "embedding_provider"
 
@@ -5093,9 +4917,7 @@ class ScimUserMapping(Base):
     __tablename__ = "scim_user_mapping"
 
     id: Mapped[int] = mapped_column(Integer, primary_key=True)
-    external_id: Mapped[str | None] = mapped_column(
-        String, unique=True, index=True, nullable=True
-    )
+    external_id: Mapped[str] = mapped_column(String, unique=True, index=True)
     user_id: Mapped[UUID] = mapped_column(
         ForeignKey("user.id", ondelete="CASCADE"), unique=True, nullable=False
     )
@@ -5152,116 +4974,3 @@ class CodeInterpreterServer(Base):
 
     id: Mapped[int] = mapped_column(Integer, primary_key=True)
     server_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
-
-
-class CacheStore(Base):
-    """Key-value cache table used by ``PostgresCacheBackend``.
-
-    Replaces Redis for simple KV caching, locks, and list operations
-    when ``CACHE_BACKEND=postgres`` (NO_VECTOR_DB deployments).
-
-    Intentionally separate from ``KVStore``:
-    - Stores raw bytes (LargeBinary) vs JSONB, matching Redis semantics.
-    - Has ``expires_at`` for TTL; rows are periodically garbage-collected.
-    - Holds ephemeral data (tokens, stop signals, lock state) not
-      persistent application config, so cleanup can be aggressive.
-    """
-
-    __tablename__ = "cache_store"
-
-    key: Mapped[str] = mapped_column(String, primary_key=True)
-    value: Mapped[bytes | None] = mapped_column(LargeBinary, nullable=True)
-    expires_at: Mapped[datetime.datetime | None] = mapped_column(
-        DateTime(timezone=True), nullable=True
-    )
-
-
-class Hook(Base):
-    """Pairs a HookPoint with a customer-provided API endpoint.
-
-    At most one Hook per HookPoint can be active at a time, enforced by a
-    partial unique index on (hook_point) where is_active=true AND deleted=false.
-    """
-
-    __tablename__ = "hook"
-
-    id: Mapped[int] = mapped_column(Integer, primary_key=True)
-    name: Mapped[str] = mapped_column(String, nullable=False)
-    hook_point: Mapped[HookPoint] = mapped_column(
-        Enum(HookPoint, native_enum=False), nullable=False
-    )
-    endpoint_url: Mapped[str | None] = mapped_column(Text, nullable=True)
-    api_key: Mapped[SensitiveValue[str] | None] = mapped_column(
-        EncryptedString(), nullable=True
-    )
-    is_reachable: Mapped[bool | None] = mapped_column(
-        Boolean, nullable=True, default=None
-    )  # null = never validated, true = last check passed, false = last check failed
-    fail_strategy: Mapped[HookFailStrategy] = mapped_column(
-        Enum(HookFailStrategy, native_enum=False),
-        nullable=False,
-        default=HookFailStrategy.HARD,
-        server_default=HookFailStrategy.HARD.value,
-    )
-    timeout_seconds: Mapped[float] = mapped_column(
-        Float, nullable=False, default=30.0, server_default="30.0"
-    )
-    is_active: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
-    deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
-    creator_id: Mapped[UUID | None] = mapped_column(
-        PGUUID(as_uuid=True),
-        ForeignKey("user.id", ondelete="SET NULL"),
-        nullable=True,
-    )
-    created_at: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True), server_default=func.now(), nullable=False
-    )
-    updated_at: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True),
-        server_default=func.now(),
-        onupdate=func.now(),
-        nullable=False,
-    )
-
-    creator: Mapped["User | None"] = relationship("User", foreign_keys=[creator_id])
-    execution_logs: Mapped[list["HookExecutionLog"]] = relationship(
-        "HookExecutionLog", back_populates="hook", cascade="all, delete-orphan"
-    )
-
-    __table_args__ = (
-        Index(
-            "ix_hook_one_active_per_point",
-            "hook_point",
-            unique=True,
-            postgresql_where=(is_active == True) & (deleted == False),  # noqa: E712
-        ),
-    )
-
-
-class HookExecutionLog(Base):
-    """Records each failed hook execution for health monitoring and debugging.
-
-    Only failures are logged. Retention: rows older than 30 days are deleted
-    by a nightly Celery task.
-    """
-
-    __tablename__ = "hook_execution_log"
-
-    id: Mapped[int] = mapped_column(Integer, primary_key=True)
-    hook_id: Mapped[int] = mapped_column(
-        Integer,
-        ForeignKey("hook.id", ondelete="CASCADE"),
-        nullable=False,
-        index=True,
-    )
-    hook_point: Mapped[HookPoint] = mapped_column(
-        Enum(HookPoint, native_enum=False), nullable=False
-    )  # denormalized for query convenience
-    error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
-    status_code: Mapped[int | None] = mapped_column(Integer, nullable=True)
-    duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
-    created_at: Mapped[datetime.datetime] = mapped_column(
-        DateTime(timezone=True), server_default=func.now(), nullable=False, index=True
-    )
-
-    hook: Mapped["Hook"] = relationship("Hook", back_populates="execution_logs")

backend/onyx/db/persona.py

@@ -205,9 +205,7 @@ def update_persona_access(
 
     NOTE: Callers are responsible for committing."""
 
-    needs_sync = False
     if is_public is not None:
-        needs_sync = True
         persona = db_session.query(Persona).filter(Persona.id == persona_id).first()
         if persona:
             persona.is_public = is_public
@@ -215,7 +213,6 @@ def update_persona_access(
     # NOTE: For user-ids and group-ids, `None` means "leave unchanged", `[]` means "clear all shares",
     # and a non-empty list means "replace with these shares".
     if user_ids is not None:
-        needs_sync = True
         db_session.query(Persona__User).filter(
             Persona__User.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -236,7 +233,6 @@ def update_persona_access(
     # MIT doesn't support group-based sharing, so we allow clearing (no-op since
     # there shouldn't be any) but raise an error if trying to add actual groups.
     if group_ids is not None:
-        needs_sync = True
         db_session.query(Persona__UserGroup).filter(
             Persona__UserGroup.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -244,10 +240,6 @@ def update_persona_access(
         if group_ids:
             raise NotImplementedError("Onyx MIT does not support group-based sharing")
 
-    # When sharing changes, user file ACLs need to be updated in the vector DB
-    if needs_sync:
-        mark_persona_user_files_for_sync(persona_id, db_session)
-
 
 def create_update_persona(
     persona_id: int | None,
@@ -859,24 +851,6 @@ def update_personas_display_priority(
         db_session.commit()
 
 
-def mark_persona_user_files_for_sync(
-    persona_id: int,
-    db_session: Session,
-) -> None:
-    """When persona sharing changes, mark all of its user files for sync
-    so that their ACLs get updated in the vector DB."""
-    persona = (
-        db_session.query(Persona)
-        .options(selectinload(Persona.user_files))
-        .filter(Persona.id == persona_id)
-        .first()
-    )
-    if not persona:
-        return
-    file_ids = [uf.id for uf in persona.user_files]
-    _mark_files_need_persona_sync(db_session, file_ids)
-
-
 def _mark_files_need_persona_sync(
     db_session: Session,
     user_file_ids: list[UUID],

backend/onyx/db/projects.py

@@ -52,7 +52,7 @@ def create_user_files(
 ) -> CategorizedFilesResult:
 
     # Categorize the files
-    categorized_files = categorize_uploaded_files(files, db_session)
+    categorized_files = categorize_uploaded_files(files)
     # NOTE: At the moment, zip metadata is not used for user files.
     # Should revisit to decide whether this should be a feature.
     upload_response = upload_files(categorized_files.acceptable, FileOrigin.USER_FILE)

backend/onyx/db/rotate_encryption_key.py

@@ -1,161 +0,0 @@
-"""Rotate encryption key for all encrypted columns.
-
-Dynamically discovers all columns using EncryptedString / EncryptedJson,
-decrypts each value with the old key, and re-encrypts with the current
-ENCRYPTION_KEY_SECRET.
-
-The operation is idempotent: rows already encrypted with the current key
-are skipped. Commits are made in batches so a crash mid-rotation can be
-safely resumed by re-running.
-"""
-
-import json
-from typing import Any
-
-from sqlalchemy import LargeBinary
-from sqlalchemy import select
-from sqlalchemy import update
-from sqlalchemy.orm import Session
-
-from onyx.configs.app_configs import ENCRYPTION_KEY_SECRET
-from onyx.db.models import Base
-from onyx.db.models import EncryptedJson
-from onyx.db.models import EncryptedString
-from onyx.utils.encryption import decrypt_bytes_to_string
-from onyx.utils.logger import setup_logger
-from onyx.utils.variable_functionality import global_version
-
-logger = setup_logger()
-
-_BATCH_SIZE = 500
-
-
-def _can_decrypt_with_current_key(data: bytes) -> bool:
-    """Check if data is already encrypted with the current key.
-
-    Passes the key explicitly so the fallback-to-raw-decode path in
-    _decrypt_bytes is NOT triggered — a clean success/failure signal.
-    """
-    try:
-        decrypt_bytes_to_string(data, key=ENCRYPTION_KEY_SECRET)
-        return True
-    except Exception:
-        return False
-
-
-def _discover_encrypted_columns() -> list[tuple[type, str, list[str], bool]]:
-    """Walk all ORM models and find columns using EncryptedString/EncryptedJson.
-
-    Returns list of (ModelClass, column_attr_name, [pk_attr_names], is_json).
-    """
-    results: list[tuple[type, str, list[str], bool]] = []
-
-    for mapper in Base.registry.mappers:
-        model_cls = mapper.class_
-        pk_names = [col.key for col in mapper.primary_key]
-
-        for prop in mapper.column_attrs:
-            for col in prop.columns:
-                if isinstance(col.type, EncryptedJson):
-                    results.append((model_cls, prop.key, pk_names, True))
-                elif isinstance(col.type, EncryptedString):
-                    results.append((model_cls, prop.key, pk_names, False))
-
-    return results
-
-
-def rotate_encryption_key(
-    db_session: Session,
-    old_key: str | None,
-    dry_run: bool = False,
-) -> dict[str, int]:
-    """Decrypt all encrypted columns with old_key and re-encrypt with the current key.
-
-    Args:
-        db_session: Active database session.
-        old_key: The previous encryption key. Pass None or "" if values were
-                 not previously encrypted with a key.
-        dry_run: If True, count rows that need rotation without modifying data.
-
-    Returns:
-        Dict of "table.column" -> number of rows re-encrypted (or would be).
-
-    Commits every _BATCH_SIZE rows so that locks are held briefly and progress
-    is preserved on crash. Already-rotated rows are detected and skipped,
-    making the operation safe to re-run.
-    """
-    if not global_version.is_ee_version():
-        raise RuntimeError("EE mode is not enabled — rotation requires EE encryption.")
-
-    if not ENCRYPTION_KEY_SECRET:
-        raise RuntimeError(
-            "ENCRYPTION_KEY_SECRET is not set — cannot rotate. "
-            "Set the target encryption key in the environment before running."
-        )
-
-    encrypted_columns = _discover_encrypted_columns()
-    totals: dict[str, int] = {}
-
-    for model_cls, col_name, pk_names, is_json in encrypted_columns:
-        table_name: str = model_cls.__tablename__  # type: ignore[attr-defined]
-        col_attr = getattr(model_cls, col_name)
-        pk_attrs = [getattr(model_cls, pk) for pk in pk_names]
-
-        # Read raw bytes directly, bypassing the TypeDecorator
-        raw_col = col_attr.property.columns[0]
-
-        stmt = select(*pk_attrs, raw_col.cast(LargeBinary)).where(col_attr.is_not(None))
-        rows = db_session.execute(stmt).all()
-
-        reencrypted = 0
-        batch_pending = 0
-        for row in rows:
-            raw_bytes: bytes | None = row[-1]
-            if raw_bytes is None:
-                continue
-
-            if _can_decrypt_with_current_key(raw_bytes):
-                continue
-
-            try:
-                if not old_key:
-                    decrypted_str = raw_bytes.decode("utf-8")
-                else:
-                    decrypted_str = decrypt_bytes_to_string(raw_bytes, key=old_key)
-
-                # For EncryptedJson, parse back to dict so the TypeDecorator
-                # can json.dumps() it cleanly (avoids double-encoding).
-                value: Any = json.loads(decrypted_str) if is_json else decrypted_str
-            except (ValueError, UnicodeDecodeError) as e:
-                pk_vals = [row[i] for i in range(len(pk_names))]
-                logger.warning(
-                    f"Could not decrypt/parse {table_name}.{col_name} "
-                    f"row {pk_vals} — skipping: {e}"
-                )
-                continue
-
-            if not dry_run:
-                pk_filters = [pk_attr == row[i] for i, pk_attr in enumerate(pk_attrs)]
-                update_stmt = (
-                    update(model_cls).where(*pk_filters).values({col_name: value})
-                )
-                db_session.execute(update_stmt)
-                batch_pending += 1
-
-                if batch_pending >= _BATCH_SIZE:
-                    db_session.commit()
-                    batch_pending = 0
-            reencrypted += 1
-
-        # Flush remaining rows in this column
-        if batch_pending > 0:
-            db_session.commit()
-
-        if reencrypted > 0:
-            totals[f"{table_name}.{col_name}"] = reencrypted
-            logger.info(
-                f"{'[DRY RUN] Would re-encrypt' if dry_run else 'Re-encrypted'} "
-                f"{reencrypted} value(s) in {table_name}.{col_name}"
-            )
-
-    return totals

backend/onyx/db/search_settings.py

@@ -129,7 +129,7 @@ def get_current_search_settings(db_session: Session) -> SearchSettings:
     latest_settings = result.scalars().first()
 
     if not latest_settings:
-        raise RuntimeError("No search settings specified; DB is not in a valid state.")
+        raise RuntimeError("No search settings specified, DB is not in a valid state")
     return latest_settings