feat(opal): Add AuxiliaryTag component and resync colors.css with Figma

Add AuxiliaryTag component (green/blue/purple/amber/gray) with icon support.

Resync all color tokens in colors.css against Figma source of truth (20 fixes):
- Fix neon alpha variant naming: -60/-30 → -a60/-a30 to disambiguate from
  Figma scale levels (e.g. --neon-amber-60 is now scale Neon/Amber/60,
  --neon-amber-a60 is the 40-at-60%-opacity highlight variant)
- Add neon scale primitives for yellow, lime, cyan, sky, magenta (50, 20, 05
  for light mode; 80, 90 for dark mode)
- Fix all neon-based theme tokens (yellow, lime, cyan, sky, magenta) from
  alpha variants to correct Figma solid swatches
- Fix background-neutral-inverted-04 (grey-75→grey-60) and -03 (grey-80→grey-75)
- Fix background-tint-inverted-04 (tint-80→tint-60)
- Fix theme-gradient-00 (grey-00→grey-100)
- Fix mask-01 (alpha-grey-100→alpha-grey-00)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.claude/skills

@@ -1 +0,0 @@
-../.cursor/skills

.cursor/skills/onyx-cli/SKILL.md

@@ -1,186 +0,0 @@
----
-name: onyx-cli
-description: Query the Onyx knowledge base using the onyx-cli command. Use when the user wants to search company documents, ask questions about internal knowledge, query connected data sources, or look up information stored in Onyx.
----
-
-# Onyx CLI — Agent Tool
-
-Onyx is an enterprise search and Gen-AI platform that connects to company documents, apps, and people. The `onyx-cli` CLI provides non-interactive commands to query the Onyx knowledge base and list available agents.
-
-## Prerequisites
-
-### 1. Check if installed
-
-```bash
-which onyx-cli
-```
-
-### 2. Install (if needed)
-
-**Primary — pip:**
-
-```bash
-pip install onyx-cli
-```
-
-**From source (Go):**
-
-```bash
-cd cli && go build -o onyx-cli . && sudo mv onyx-cli /usr/local/bin/
-```
-
-### 3. Check if configured
-
-```bash
-onyx-cli validate-config
-```
-
-This checks the config file exists, API key is present, and tests the server connection via `/api/me`. Exit code 0 on success, non-zero with a descriptive error on failure.
-
-If unconfigured, you have two options:
-
-**Option A — Interactive setup (requires user input):**
-
-```bash
-onyx-cli configure
-```
-
-This prompts for the Onyx server URL and API key, tests the connection, and saves config.
-
-**Option B — Environment variables (non-interactive, preferred for agents):**
-
-```bash
-export ONYX_SERVER_URL="https://your-onyx-server.com"  # default: https://cloud.onyx.app
-export ONYX_API_KEY="your-api-key"
-```
-
-Environment variables override the config file. If these are set, no config file is needed.
-
-| Variable | Required | Description |
-|----------|----------|-------------|
-| `ONYX_SERVER_URL` | No | Onyx server base URL (default: `https://cloud.onyx.app`) |
-| `ONYX_API_KEY` | Yes | API key for authentication |
-| `ONYX_PERSONA_ID` | No | Default agent/persona ID |
-
-If neither the config file nor environment variables are set, tell the user that `onyx-cli` needs to be configured and ask them to either:
-- Run `onyx-cli configure` interactively, or
-- Set `ONYX_SERVER_URL` and `ONYX_API_KEY` environment variables
-
-## Commands
-
-### Validate configuration
-
-```bash
-onyx-cli validate-config
-```
-
-Checks config file exists, API key is present, and tests the server connection. Use this before `ask` or `agents` to confirm the CLI is properly set up.
-
-### List available agents
-
-```bash
-onyx-cli agents
-```
-
-Prints a table of agent IDs, names, and descriptions. Use `--json` for structured output:
-
-```bash
-onyx-cli agents --json
-```
-
-Use agent IDs with `ask --agent-id` to query a specific agent.
-
-### Basic query (plain text output)
-
-```bash
-onyx-cli ask "What is our company's PTO policy?"
-```
-
-Streams the answer as plain text to stdout. Exit code 0 on success, non-zero on error.
-
-### JSON output (structured events)
-
-```bash
-onyx-cli ask --json "What authentication methods do we support?"
-```
-
-Outputs JSON-encoded parsed stream events (one object per line). Key event objects include message deltas, stop, errors, search-start, and citation payloads.
-
-Each line is a JSON object with this envelope:
-
-```json
-{"type": "<event_type>", "event": { ... }}
-```
-
-| Event Type | Description |
-|------------|-------------|
-| `message_delta` | Content token — concatenate all `content` fields for the full answer |
-| `stop` | Stream complete |
-| `error` | Error with `error` message field |
-| `search_tool_start` | Onyx started searching documents |
-| `citation_info` | Source citation — see shape below |
-
-`citation_info` event shape:
-
-```json
-{
-  "type": "citation_info",
-  "event": {
-    "citation_number": 1,
-    "document_id": "abc123def456",
-    "placement": {"turn_index": 0, "tab_index": 0, "sub_turn_index": null}
-  }
-}
-```
-
-`placement` is metadata about where in the conversation the citation appeared and can be ignored for most use cases.
-
-### Specify an agent
-
-```bash
-onyx-cli ask --agent-id 5 "Summarize our Q4 roadmap"
-```
-
-Uses a specific Onyx agent/persona instead of the default.
-
-### All flags
-
-| Flag | Type | Description |
-|------|------|-------------|
-| `--agent-id` | int | Agent ID to use (overrides default) |
-| `--json` | bool | Output raw NDJSON events instead of plain text |
-
-## Statelessness
-
-Each `onyx-cli ask` call creates an independent chat session. There is no built-in way to chain context across multiple `ask` invocations — every call starts fresh. If you need multi-turn conversation with memory, use the interactive TUI (`onyx-cli` or `onyx-cli chat`) instead.
-
-## When to Use
-
-Use `onyx-cli ask` when:
-
-- The user asks about company-specific information (policies, docs, processes)
-- You need to search internal knowledge bases or connected data sources
-- The user references Onyx, asks you to "search Onyx", or wants to query their documents
-- You need context from company wikis, Confluence, Google Drive, Slack, or other connected sources
-
-Do NOT use when:
-
-- The question is about general programming knowledge (use your own knowledge)
-- The user is asking about code in the current repository (use grep/read tools)
-- The user hasn't mentioned Onyx and the question doesn't require internal company data
-
-## Examples
-
-```bash
-# Simple question
-onyx-cli ask "What are the steps to deploy to production?"
-
-# Get structured output for parsing
-onyx-cli ask --json "List all active API integrations"
-
-# Use a specialized agent
-onyx-cli ask --agent-id 3 "What were the action items from last week's standup?"
-
-# Pipe the answer into another command
-onyx-cli ask "What is the database schema for users?" | head -20
-```

.cursor/skills/playwright/SKILL.md

@@ -1,248 +0,0 @@
----
-name: playwright-e2e-tests
-description: Write and maintain Playwright end-to-end tests for the Onyx application. Use when creating new E2E tests, debugging test failures, adding test coverage, or when the user mentions Playwright, E2E tests, or browser testing.
----
-
-# Playwright E2E Tests
-
-## Project Layout
-
-- **Tests**: `web/tests/e2e/` — organized by feature (`auth/`, `admin/`, `chat/`, `assistants/`, `connectors/`, `mcp/`)
-- **Config**: `web/playwright.config.ts`
-- **Utilities**: `web/tests/e2e/utils/`
-- **Constants**: `web/tests/e2e/constants.ts`
-- **Global setup**: `web/tests/e2e/global-setup.ts`
-- **Output**: `web/output/playwright/`
-
-## Imports
-
-Always use absolute imports with the `@tests/e2e/` prefix — never relative paths (`../`, `../../`). The alias is defined in `web/tsconfig.json` and resolves to `web/tests/`.
-
-```typescript
-import { loginAs } from "@tests/e2e/utils/auth";
-import { OnyxApiClient } from "@tests/e2e/utils/onyxApiClient";
-import { TEST_ADMIN_CREDENTIALS } from "@tests/e2e/constants";
-```
-
-All new files should be `.ts`, not `.js`.
-
-## Running Tests
-
-```bash
-# Run a specific test file
-npx playwright test web/tests/e2e/chat/default_assistant.spec.ts
-
-# Run a specific project
-npx playwright test --project admin
-npx playwright test --project exclusive
-```
-
-## Test Projects
-
-| Project | Description | Parallelism |
-|---------|-------------|-------------|
-| `admin` | Standard tests (excludes `@exclusive`) | Parallel |
-| `exclusive` | Serial, slower tests (tagged `@exclusive`) | 1 worker |
-
-All tests use `admin_auth.json` storage state by default (pre-authenticated admin session).
-
-## Authentication
-
-Global setup (`global-setup.ts`) runs automatically before all tests and handles:
-
-- Server readiness check (polls health endpoint, 60s timeout)
-- Provisioning test users: admin, admin2, and a **pool of worker users** (`worker0@example.com` through `worker7@example.com`) (idempotent)
-- API login + saving storage states: `admin_auth.json`, `admin2_auth.json`, and `worker{N}_auth.json` for each worker user
-- Setting display name to `"worker"` for each worker user
-- Promoting admin2 to admin role
-- Ensuring a public LLM provider exists
-
-Both test projects set `storageState: "admin_auth.json"`, so **every test starts pre-authenticated as admin with no login code needed**.
-
-When a test needs a different user, use API-based login — never drive the login UI:
-
-```typescript
-import { loginAs } from "@tests/e2e/utils/auth";
-
-await page.context().clearCookies();
-await loginAs(page, "admin2");
-
-// Log in as the worker-specific user (preferred for test isolation):
-import { loginAsWorkerUser } from "@tests/e2e/utils/auth";
-await page.context().clearCookies();
-await loginAsWorkerUser(page, testInfo.workerIndex);
-```
-
-## Test Structure
-
-Tests start pre-authenticated as admin — navigate and test directly:
-
-```typescript
-import { test, expect } from "@playwright/test";
-
-test.describe("Feature Name", () => {
-  test("should describe expected behavior clearly", async ({ page }) => {
-    await page.goto("/app");
-    await page.waitForLoadState("networkidle");
-    // Already authenticated as admin — go straight to testing
-  });
-});
-```
-
-**User isolation** — tests that modify visible app state (creating assistants, sending chat messages, pinning items) should run as a **worker-specific user** and clean up resources in `afterAll`. Global setup provisions a pool of worker users (`worker0@example.com` through `worker7@example.com`). `loginAsWorkerUser` maps `testInfo.workerIndex` to a pool slot via modulo, so retry workers (which get incrementing indices beyond the pool size) safely reuse existing users. This ensures parallel workers never share user state, keeps usernames deterministic for screenshots, and avoids cross-contamination:
-
-```typescript
-import { test } from "@playwright/test";
-import { loginAsWorkerUser } from "@tests/e2e/utils/auth";
-
-test.beforeEach(async ({ page }, testInfo) => {
-  await page.context().clearCookies();
-  await loginAsWorkerUser(page, testInfo.workerIndex);
-});
-```
-
-If the test requires admin privileges *and* modifies visible state, use `"admin2"` instead — it's a pre-provisioned admin account that keeps the primary `"admin"` clean for other parallel tests. Switch to `"admin"` only for privileged setup (creating providers, configuring tools), then back to the worker user for the actual test. See `chat/default_assistant.spec.ts` for a full example.
-
-`loginAsRandomUser` exists for the rare case where the test requires a brand-new user (e.g. onboarding flows). Avoid it elsewhere — it produces non-deterministic usernames that complicate screenshots.
-
-**API resource setup** — only when tests need to create backend resources (image gen configs, web search providers, MCP servers). Use `beforeAll`/`afterAll` with `OnyxApiClient` to create and clean up. See `chat/default_assistant.spec.ts` or `mcp/mcp_oauth_flow.spec.ts` for examples. This is uncommon (~4 of 37 test files).
-
-## Key Utilities
-
-### `OnyxApiClient` (`@tests/e2e/utils/onyxApiClient`)
-
-Backend API client for test setup/teardown. Key methods:
-
-- **Connectors**: `createFileConnector()`, `deleteCCPair()`, `pauseConnector()`
-- **LLM Providers**: `ensurePublicProvider()`, `createRestrictedProvider()`, `setProviderAsDefault()`
-- **Assistants**: `createAssistant()`, `deleteAssistant()`, `findAssistantByName()`
-- **User Groups**: `createUserGroup()`, `deleteUserGroup()`, `setUserRole()`
-- **Tools**: `createWebSearchProvider()`, `createImageGenerationConfig()`
-- **Chat**: `createChatSession()`, `deleteChatSession()`
-
-### `chatActions` (`@tests/e2e/utils/chatActions`)
-
-- `sendMessage(page, message)` — sends a message and waits for AI response
-- `startNewChat(page)` — clicks new-chat button and waits for intro
-- `verifyDefaultAssistantIsChosen(page)` — checks Onyx logo is visible
-- `verifyAssistantIsChosen(page, name)` — checks assistant name display
-- `switchModel(page, modelName)` — switches LLM model via popover
-
-### `visualRegression` (`@tests/e2e/utils/visualRegression`)
-
-- `expectScreenshot(page, { name, mask?, hide?, fullPage? })`
-- `expectElementScreenshot(locator, { name, mask?, hide? })`
-- Controlled by `VISUAL_REGRESSION=true` env var
-
-### `theme` (`@tests/e2e/utils/theme`)
-
-- `THEMES` — `["light", "dark"] as const` array for iterating over both themes
-- `setThemeBeforeNavigation(page, theme)` — sets `next-themes` theme via `localStorage` before navigation
-
-When tests need light/dark screenshots, loop over `THEMES` at the `test.describe` level and call `setThemeBeforeNavigation` in `beforeEach` **before** any `page.goto()`. Include the theme in screenshot names. See `admin/admin_pages.spec.ts` or `chat/chat_message_rendering.spec.ts` for examples:
-
-```typescript
-import { THEMES, setThemeBeforeNavigation } from "@tests/e2e/utils/theme";
-
-for (const theme of THEMES) {
-  test.describe(`Feature (${theme} mode)`, () => {
-    test.beforeEach(async ({ page }) => {
-      await setThemeBeforeNavigation(page, theme);
-    });
-
-    test("renders correctly", async ({ page }) => {
-      await page.goto("/app");
-      await expectScreenshot(page, { name: `feature-${theme}` });
-    });
-  });
-}
-```
-
-### `tools` (`@tests/e2e/utils/tools`)
-
-- `TOOL_IDS` — centralized `data-testid` selectors for tool options
-- `openActionManagement(page)` — opens the tool management popover
-
-## Locator Strategy
-
-Use locators in this priority order:
-
-1. **`data-testid` / `aria-label`** — preferred for Onyx components
-   ```typescript
-   page.getByTestId("AppSidebar/new-session")
-   page.getByLabel("admin-page-title")
-   ```
-
-2. **Role-based** — for standard HTML elements
-   ```typescript
-   page.getByRole("button", { name: "Create" })
-   page.getByRole("dialog")
-   ```
-
-3. **Text/Label** — for visible text content
-   ```typescript
-   page.getByText("Custom Assistant")
-   page.getByLabel("Email")
-   ```
-
-4. **CSS selectors** — last resort, only when above won't work
-   ```typescript
-   page.locator('input[name="name"]')
-   page.locator("#onyx-chat-input-textarea")
-   ```
-
-**Never use** `page.locator` with complex CSS/XPath when a built-in locator works.
-
-## Assertions
-
-Use web-first assertions — they auto-retry until the condition is met:
-
-```typescript
-// Visibility
-await expect(page.getByTestId("onyx-logo")).toBeVisible({ timeout: 5000 });
-
-// Text content
-await expect(page.getByTestId("assistant-name-display")).toHaveText("My Assistant");
-
-// Count
-await expect(page.locator('[data-testid="onyx-ai-message"]')).toHaveCount(2, { timeout: 30000 });
-
-// URL
-await expect(page).toHaveURL(/chatId=/);
-
-// Element state
-await expect(toggle).toBeChecked();
-await expect(button).toBeEnabled();
-```
-
-**Never use** `assert` statements or hardcoded `page.waitForTimeout()`.
-
-## Waiting Strategy
-
-```typescript
-// Wait for load state after navigation
-await page.goto("/app");
-await page.waitForLoadState("networkidle");
-
-// Wait for specific element
-await page.getByTestId("chat-intro").waitFor({ state: "visible", timeout: 10000 });
-
-// Wait for URL change
-await page.waitForFunction(() => window.location.href.includes("chatId="), null, { timeout: 10000 });
-
-// Wait for network response
-await page.waitForResponse(resp => resp.url().includes("/api/chat") && resp.status() === 200);
-```
-
-## Best Practices
-
-1. **Descriptive test names** — clearly state expected behavior: `"should display greeting message when opening new chat"`
-2. **API-first setup** — use `OnyxApiClient` for backend state; reserve UI interactions for the behavior under test
-3. **User isolation** — tests that modify visible app state (sidebar, chat history) should run as the worker-specific user via `loginAsWorkerUser(page, testInfo.workerIndex)` (not admin) and clean up resources in `afterAll`. Each parallel worker gets its own user, preventing cross-contamination. Reserve `loginAsRandomUser` for flows that require a brand-new user (e.g. onboarding)
-4. **DRY helpers** — extract reusable logic into `utils/` with JSDoc comments
-5. **No hardcoded waits** — use `waitFor`, `waitForLoadState`, or web-first assertions
-6. **Parallel-safe** — no shared mutable state between tests. Prefer static, human-readable names (e.g. `"E2E-CMD Chat 1"`) and clean up resources by ID in `afterAll`. This keeps screenshots deterministic and avoids needing to mask/hide dynamic text. Only fall back to timestamps (`\`test-${Date.now()}\``) when resources cannot be reliably cleaned up or when name collisions across parallel workers would cause functional failures
-7. **Error context** — catch and re-throw with useful debug info (page text, URL, etc.)
-8. **Tag slow tests** — mark serial/slow tests with `@exclusive` in the test title
-9. **Visual regression** — use `expectScreenshot()` for UI consistency checks
-10. **Minimal comments** — only comment to clarify non-obvious intent; never restate what the next line of code does

.github/CODEOWNERS

@@ -8,6 +8,3 @@
 # Agent context files
 /CLAUDE.md @Weves
 /AGENTS.md @Weves
-
-# Beta cherry-pick workflow owners
-/.github/workflows/post-merge-beta-cherry-pick.yml @justin-tahara @jmelahman

.github/actions/build-backend-image/action.yml

@@ -1,73 +0,0 @@
-name: "Build Backend Image"
-description: "Builds and pushes the backend Docker image with cache reuse"
-inputs:
-  runs-on-ecr-cache:
-    description: "ECR cache registry from runs-on/action"
-    required: true
-  ref-name:
-    description: "Git ref name used for cache suffix fallback"
-    required: true
-  pr-number:
-    description: "Optional PR number for cache suffix"
-    required: false
-    default: ""
-  github-sha:
-    description: "Commit SHA used for cache keys"
-    required: true
-  run-id:
-    description: "GitHub run ID used in output image tag"
-    required: true
-  docker-username:
-    description: "Docker Hub username"
-    required: true
-  docker-token:
-    description: "Docker Hub token"
-    required: true
-  docker-no-cache:
-    description: "Set to 'true' to disable docker build cache"
-    required: false
-    default: "false"
-runs:
-  using: "composite"
-  steps:
-    - name: Format branch name for cache
-      id: format-branch
-      shell: bash
-      env:
-        PR_NUMBER: ${{ inputs.pr-number }}
-        REF_NAME: ${{ inputs.ref-name }}
-      run: |
-        if [ -n "${PR_NUMBER}" ]; then
-          CACHE_SUFFIX="${PR_NUMBER}"
-        else
-          # shellcheck disable=SC2001
-          CACHE_SUFFIX=$(echo "${REF_NAME}" | sed 's/[^A-Za-z0-9._-]/-/g')
-        fi
-        echo "cache-suffix=${CACHE_SUFFIX}" >> "$GITHUB_OUTPUT"
-
-    - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # ratchet:docker/setup-buildx-action@v3
-
-    - name: Login to Docker Hub
-      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-      with:
-        username: ${{ inputs.docker-username }}
-        password: ${{ inputs.docker-token }}
-
-    - name: Build and push Backend Docker image
-      uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # ratchet:docker/build-push-action@v6
-      with:
-        context: ./backend
-        file: ./backend/Dockerfile
-        push: true
-        tags: ${{ inputs.runs-on-ecr-cache }}:nightly-llm-it-backend-${{ inputs.run-id }}
-        cache-from: |
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache-${{ inputs.github-sha }}
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache-${{ steps.format-branch.outputs.cache-suffix }}
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache
-          type=registry,ref=onyxdotapp/onyx-backend:latest
-        cache-to: |
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache-${{ inputs.github-sha }},mode=max
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache-${{ steps.format-branch.outputs.cache-suffix }},mode=max
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:backend-cache,mode=max
-        no-cache: ${{ inputs.docker-no-cache == 'true' }}

.github/actions/build-integration-image/action.yml

@@ -1,76 +0,0 @@
-name: "Build Integration Image"
-description: "Builds and pushes the integration test image with docker bake"
-inputs:
-  runs-on-ecr-cache:
-    description: "ECR cache registry from runs-on/action"
-    required: true
-  ref-name:
-    description: "Git ref name used for cache suffix fallback"
-    required: true
-  pr-number:
-    description: "Optional PR number for cache suffix"
-    required: false
-    default: ""
-  github-sha:
-    description: "Commit SHA used for cache keys"
-    required: true
-  run-id:
-    description: "GitHub run ID used in output image tag"
-    required: true
-  docker-username:
-    description: "Docker Hub username"
-    required: true
-  docker-token:
-    description: "Docker Hub token"
-    required: true
-runs:
-  using: "composite"
-  steps:
-    - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # ratchet:docker/setup-buildx-action@v3
-
-    - name: Login to Docker Hub
-      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-      with:
-        username: ${{ inputs.docker-username }}
-        password: ${{ inputs.docker-token }}
-
-    - name: Format branch name for cache
-      id: format-branch
-      shell: bash
-      env:
-        PR_NUMBER: ${{ inputs.pr-number }}
-        REF_NAME: ${{ inputs.ref-name }}
-      run: |
-        if [ -n "${PR_NUMBER}" ]; then
-          CACHE_SUFFIX="${PR_NUMBER}"
-        else
-          # shellcheck disable=SC2001
-          CACHE_SUFFIX=$(echo "${REF_NAME}" | sed 's/[^A-Za-z0-9._-]/-/g')
-        fi
-        echo "cache-suffix=${CACHE_SUFFIX}" >> "$GITHUB_OUTPUT"
-
-    - name: Build and push integration test image with Docker Bake
-      shell: bash
-      env:
-        RUNS_ON_ECR_CACHE: ${{ inputs.runs-on-ecr-cache }}
-        INTEGRATION_REPOSITORY: ${{ inputs.runs-on-ecr-cache }}
-        TAG: nightly-llm-it-${{ inputs.run-id }}
-        CACHE_SUFFIX: ${{ steps.format-branch.outputs.cache-suffix }}
-        HEAD_SHA: ${{ inputs.github-sha }}
-      run: |
-        docker buildx bake --push \
-          --set backend.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache-${HEAD_SHA} \
-          --set backend.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache-${CACHE_SUFFIX} \
-          --set backend.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache \
-          --set backend.cache-from=type=registry,ref=onyxdotapp/onyx-backend:latest \
-          --set backend.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache-${HEAD_SHA},mode=max \
-          --set backend.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache-${CACHE_SUFFIX},mode=max \
-          --set backend.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:backend-cache,mode=max \
-          --set integration.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache-${HEAD_SHA} \
-          --set integration.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache-${CACHE_SUFFIX} \
-          --set integration.cache-from=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache \
-          --set integration.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache-${HEAD_SHA},mode=max \
-          --set integration.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache-${CACHE_SUFFIX},mode=max \
-          --set integration.cache-to=type=registry,ref=${RUNS_ON_ECR_CACHE}:integration-cache,mode=max \
-          integration

.github/actions/build-model-server-image/action.yml

@@ -1,68 +0,0 @@
-name: "Build Model Server Image"
-description: "Builds and pushes the model server Docker image with cache reuse"
-inputs:
-  runs-on-ecr-cache:
-    description: "ECR cache registry from runs-on/action"
-    required: true
-  ref-name:
-    description: "Git ref name used for cache suffix fallback"
-    required: true
-  pr-number:
-    description: "Optional PR number for cache suffix"
-    required: false
-    default: ""
-  github-sha:
-    description: "Commit SHA used for cache keys"
-    required: true
-  run-id:
-    description: "GitHub run ID used in output image tag"
-    required: true
-  docker-username:
-    description: "Docker Hub username"
-    required: true
-  docker-token:
-    description: "Docker Hub token"
-    required: true
-runs:
-  using: "composite"
-  steps:
-    - name: Format branch name for cache
-      id: format-branch
-      shell: bash
-      env:
-        PR_NUMBER: ${{ inputs.pr-number }}
-        REF_NAME: ${{ inputs.ref-name }}
-      run: |
-        if [ -n "${PR_NUMBER}" ]; then
-          CACHE_SUFFIX="${PR_NUMBER}"
-        else
-          # shellcheck disable=SC2001
-          CACHE_SUFFIX=$(echo "${REF_NAME}" | sed 's/[^A-Za-z0-9._-]/-/g')
-        fi
-        echo "cache-suffix=${CACHE_SUFFIX}" >> "$GITHUB_OUTPUT"
-
-    - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # ratchet:docker/setup-buildx-action@v3
-
-    - name: Login to Docker Hub
-      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-      with:
-        username: ${{ inputs.docker-username }}
-        password: ${{ inputs.docker-token }}
-
-    - name: Build and push Model Server Docker image
-      uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # ratchet:docker/build-push-action@v6
-      with:
-        context: ./backend
-        file: ./backend/Dockerfile.model_server
-        push: true
-        tags: ${{ inputs.runs-on-ecr-cache }}:nightly-llm-it-model-server-${{ inputs.run-id }}
-        cache-from: |
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache-${{ inputs.github-sha }}
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache-${{ steps.format-branch.outputs.cache-suffix }}
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache
-          type=registry,ref=onyxdotapp/onyx-model-server:latest
-        cache-to: |
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache-${{ inputs.github-sha }},mode=max
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache-${{ steps.format-branch.outputs.cache-suffix }},mode=max
-          type=registry,ref=${{ inputs.runs-on-ecr-cache }}:model-server-cache,mode=max

.github/actions/run-nightly-provider-chat-test/action.yml

@@ -1,130 +0,0 @@
-name: "Run Nightly Provider Chat Test"
-description: "Starts required compose services and runs nightly provider integration test"
-inputs:
-  provider:
-    description: "Provider slug for NIGHTLY_LLM_PROVIDER"
-    required: true
-  models:
-    description: "Comma-separated model list for NIGHTLY_LLM_MODELS"
-    required: true
-  provider-api-key:
-    description: "API key for NIGHTLY_LLM_API_KEY"
-    required: false
-    default: ""
-  strict:
-    description: "String true/false for NIGHTLY_LLM_STRICT"
-    required: true
-  api-base:
-    description: "Optional NIGHTLY_LLM_API_BASE"
-    required: false
-    default: ""
-  api-version:
-    description: "Optional NIGHTLY_LLM_API_VERSION"
-    required: false
-    default: ""
-  deployment-name:
-    description: "Optional NIGHTLY_LLM_DEPLOYMENT_NAME"
-    required: false
-    default: ""
-  custom-config-json:
-    description: "Optional NIGHTLY_LLM_CUSTOM_CONFIG_JSON"
-    required: false
-    default: ""
-  runs-on-ecr-cache:
-    description: "ECR cache registry from runs-on/action"
-    required: true
-  run-id:
-    description: "GitHub run ID used in image tags"
-    required: true
-  docker-username:
-    description: "Docker Hub username"
-    required: true
-  docker-token:
-    description: "Docker Hub token"
-    required: true
-runs:
-  using: "composite"
-  steps:
-    - name: Login to Docker Hub
-      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-      with:
-        username: ${{ inputs.docker-username }}
-        password: ${{ inputs.docker-token }}
-
-    - name: Create .env file for Docker Compose
-      shell: bash
-      env:
-        ECR_CACHE: ${{ inputs.runs-on-ecr-cache }}
-        RUN_ID: ${{ inputs.run-id }}
-      run: |
-        cat <<EOF2 > deployment/docker_compose/.env
-        COMPOSE_PROFILES=s3-filestore
-        ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
-        LICENSE_ENFORCEMENT_ENABLED=false
-        AUTH_TYPE=basic
-        POSTGRES_POOL_PRE_PING=true
-        POSTGRES_USE_NULL_POOL=true
-        REQUIRE_EMAIL_VERIFICATION=false
-        DISABLE_TELEMETRY=true
-        INTEGRATION_TESTS_MODE=true
-        AUTO_LLM_UPDATE_INTERVAL_SECONDS=10
-        AWS_REGION_NAME=us-west-2
-        ONYX_BACKEND_IMAGE=${ECR_CACHE}:nightly-llm-it-backend-${RUN_ID}
-        ONYX_MODEL_SERVER_IMAGE=${ECR_CACHE}:nightly-llm-it-model-server-${RUN_ID}
-        EOF2
-
-    - name: Start Docker containers
-      shell: bash
-      run: |
-        cd deployment/docker_compose
-        docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --wait \
-          relational_db \
-          index \
-          cache \
-          minio \
-          api_server \
-          inference_model_server
-
-    - name: Run nightly provider integration test
-      uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # ratchet:nick-fields/retry@v3
-      env:
-        MODELS: ${{ inputs.models }}
-        NIGHTLY_LLM_PROVIDER: ${{ inputs.provider }}
-        NIGHTLY_LLM_API_KEY: ${{ inputs.provider-api-key }}
-        NIGHTLY_LLM_API_BASE: ${{ inputs.api-base }}
-        NIGHTLY_LLM_API_VERSION: ${{ inputs.api-version }}
-        NIGHTLY_LLM_DEPLOYMENT_NAME: ${{ inputs.deployment-name }}
-        NIGHTLY_LLM_CUSTOM_CONFIG_JSON: ${{ inputs.custom-config-json }}
-        NIGHTLY_LLM_STRICT: ${{ inputs.strict }}
-        RUNS_ON_ECR_CACHE: ${{ inputs.runs-on-ecr-cache }}
-        RUN_ID: ${{ inputs.run-id }}
-      with:
-        timeout_minutes: 20
-        max_attempts: 2
-        retry_wait_seconds: 10
-        command: |
-          docker run --rm --network onyx_default \
-            --name test-runner \
-            -e POSTGRES_HOST=relational_db \
-            -e POSTGRES_USER=postgres \
-            -e POSTGRES_PASSWORD=password \
-            -e POSTGRES_DB=postgres \
-            -e DB_READONLY_USER=db_readonly_user \
-            -e DB_READONLY_PASSWORD=password \
-            -e POSTGRES_POOL_PRE_PING=true \
-            -e POSTGRES_USE_NULL_POOL=true \
-            -e VESPA_HOST=index \
-            -e REDIS_HOST=cache \
-            -e API_SERVER_HOST=api_server \
-            -e TEST_WEB_HOSTNAME=test-runner \
-            -e AWS_REGION_NAME=us-west-2 \
-            -e NIGHTLY_LLM_PROVIDER="${NIGHTLY_LLM_PROVIDER}" \
-            -e NIGHTLY_LLM_MODELS="${MODELS}" \
-            -e NIGHTLY_LLM_API_KEY="${NIGHTLY_LLM_API_KEY}" \
-            -e NIGHTLY_LLM_API_BASE="${NIGHTLY_LLM_API_BASE}" \
-            -e NIGHTLY_LLM_API_VERSION="${NIGHTLY_LLM_API_VERSION}" \
-            -e NIGHTLY_LLM_DEPLOYMENT_NAME="${NIGHTLY_LLM_DEPLOYMENT_NAME}" \
-            -e NIGHTLY_LLM_CUSTOM_CONFIG_JSON="${NIGHTLY_LLM_CUSTOM_CONFIG_JSON}" \
-            -e NIGHTLY_LLM_STRICT="${NIGHTLY_LLM_STRICT}" \
-            ${RUNS_ON_ECR_CACHE}:nightly-llm-it-${RUN_ID} \
-            /app/tests/integration/tests/llm_workflows/test_nightly_provider_chat_workflow.py

.github/actions/slack-notify/action.yml

@@ -1,14 +1,11 @@
-name: "Slack Notify"
-description: "Sends a Slack notification for workflow events"
+name: "Slack Notify on Failure"
+description: "Sends a Slack notification when a workflow fails"
 inputs:
   webhook-url:
     description: "Slack webhook URL (can also use SLACK_WEBHOOK_URL env var)"
     required: false
-  details:
-    description: "Additional message body content"
-    required: false
   failed-jobs:
-    description: "Deprecated alias for details"
+    description: "List of failed job names (newline-separated)"
     required: false
   title:
     description: "Title for the notification"
@@ -24,7 +21,6 @@ runs:
       shell: bash
       env:
         SLACK_WEBHOOK_URL: ${{ inputs.webhook-url }}
-        DETAILS: ${{ inputs.details }}
         FAILED_JOBS: ${{ inputs.failed-jobs }}
         TITLE: ${{ inputs.title }}
         REF_NAME: ${{ inputs.ref-name }}
@@ -48,18 +44,6 @@ runs:
           REF_NAME="$GITHUB_REF_NAME"
         fi
 
-        if [ -z "$DETAILS" ]; then
-          DETAILS="$FAILED_JOBS"
-        fi
-
-        normalize_multiline() {
-          printf '%s' "$1" | awk 'BEGIN { ORS=""; first=1 } { if (!first) printf "\\n"; printf "%s", $0; first=0 }'
-        }
-
-        DETAILS="$(normalize_multiline "$DETAILS")"
-        REF_NAME="$(normalize_multiline "$REF_NAME")"
-        TITLE="$(normalize_multiline "$TITLE")"
-
         # Escape JSON special characters
         escape_json() {
           local input="$1"
@@ -75,12 +59,12 @@ runs:
         }
 
         REF_NAME_ESC=$(escape_json "$REF_NAME")
-        DETAILS_ESC=$(escape_json "$DETAILS")
+        FAILED_JOBS_ESC=$(escape_json "$FAILED_JOBS")
         WORKFLOW_URL_ESC=$(escape_json "$WORKFLOW_URL")
         TITLE_ESC=$(escape_json "$TITLE")
 
         # Build JSON payload piece by piece
-        # Note: DETAILS_ESC already contains \n sequences that should remain as \n in JSON
+        # Note: FAILED_JOBS_ESC already contains \n sequences that should remain as \n in JSON
         PAYLOAD="{"
         PAYLOAD="${PAYLOAD}\"text\":\"${TITLE_ESC}\","
         PAYLOAD="${PAYLOAD}\"blocks\":[{"
@@ -95,10 +79,10 @@ runs:
         PAYLOAD="${PAYLOAD}{\"type\":\"mrkdwn\",\"text\":\"*Run ID:*\\n#${RUN_NUMBER}\"}"
         PAYLOAD="${PAYLOAD}]"
         PAYLOAD="${PAYLOAD}}"
-        if [ -n "$DETAILS" ]; then
+        if [ -n "$FAILED_JOBS" ]; then
           PAYLOAD="${PAYLOAD},{"
           PAYLOAD="${PAYLOAD}\"type\":\"section\","
-          PAYLOAD="${PAYLOAD}\"text\":{\"type\":\"mrkdwn\",\"text\":\"${DETAILS_ESC}\"}"
+          PAYLOAD="${PAYLOAD}\"text\":{\"type\":\"mrkdwn\",\"text\":\"*Failed Jobs:*\\n${FAILED_JOBS_ESC}\"}"
           PAYLOAD="${PAYLOAD}}"
         fi
         PAYLOAD="${PAYLOAD},{"
@@ -115,3 +99,4 @@ runs:
         curl -X POST -H 'Content-type: application/json' \
           --data "$PAYLOAD" \
           "$SLACK_WEBHOOK_URL"
+

.github/pull_request_template.md

@@ -8,5 +8,5 @@
 
 ## Additional Options
 
-- [ ] [Optional] Please cherry-pick this PR to the latest release version.
+- [ ] [Required] I have considered whether this PR needs to be cherry-picked to the latest beta branch.
 - [ ] [Optional] Override Linear Check

.github/workflows/deployment.yml

@@ -29,32 +29,20 @@ jobs:
       build-backend-craft: ${{ steps.check.outputs.build-backend-craft }}
       build-model-server: ${{ steps.check.outputs.build-model-server }}
       is-cloud-tag: ${{ steps.check.outputs.is-cloud-tag }}
+      is-stable: ${{ steps.check.outputs.is-stable }}
       is-beta: ${{ steps.check.outputs.is-beta }}
+      is-stable-standalone: ${{ steps.check.outputs.is-stable-standalone }}
       is-beta-standalone: ${{ steps.check.outputs.is-beta-standalone }}
-      is-latest: ${{ steps.check.outputs.is-latest }}
+      is-craft-latest: ${{ steps.check.outputs.is-craft-latest }}
       is-test-run: ${{ steps.check.outputs.is-test-run }}
       sanitized-tag: ${{ steps.check.outputs.sanitized-tag }}
       short-sha: ${{ steps.check.outputs.short-sha }}
     steps:
-      - name: Checkout (for git tags)
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-          fetch-depth: 0
-          fetch-tags: true
-
-      - name: Setup uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          version: "0.9.9"
-          enable-cache: false
-
       - name: Check which components to build and version info
         id: check
         env:
           EVENT_NAME: ${{ github.event_name }}
         run: |
-          set -eo pipefail
           TAG="${GITHUB_REF_NAME}"
           # Sanitize tag name by replacing slashes with hyphens (for Docker tag compatibility)
           SANITIZED_TAG=$(echo "$TAG" | tr '/' '-')
@@ -66,8 +54,9 @@ jobs:
           IS_VERSION_TAG=false
           IS_STABLE=false
           IS_BETA=false
+          IS_STABLE_STANDALONE=false
           IS_BETA_STANDALONE=false
-          IS_LATEST=false
+          IS_CRAFT_LATEST=false
           IS_PROD_TAG=false
           IS_TEST_RUN=false
           BUILD_DESKTOP=false
@@ -78,6 +67,9 @@ jobs:
           BUILD_MODEL_SERVER=true
 
           # Determine tag type based on pattern matching (do regex checks once)
+          if [[ "$TAG" == craft-* ]]; then
+            IS_CRAFT_LATEST=true
+          fi
           if [[ "$TAG" == *cloud* ]]; then
             IS_CLOUD=true
           fi
@@ -105,28 +97,20 @@ jobs:
             fi
           fi
 
+          # Craft-latest builds backend with Craft enabled
+          if [[ "$IS_CRAFT_LATEST" == "true" ]]; then
+            BUILD_BACKEND_CRAFT=true
+            BUILD_BACKEND=false
+          fi
+
           # Standalone version checks (for backend/model-server - version excluding cloud tags)
+          if [[ "$IS_STABLE" == "true" ]] && [[ "$IS_CLOUD" != "true" ]]; then
+            IS_STABLE_STANDALONE=true
+          fi
           if [[ "$IS_BETA" == "true" ]] && [[ "$IS_CLOUD" != "true" ]]; then
             IS_BETA_STANDALONE=true
           fi
 
-          # Determine if this tag should get the "latest" Docker tag.
-          # Only the highest semver stable tag (vX.Y.Z exactly) gets "latest".
-          if [[ "$IS_STABLE" == "true" ]]; then
-            HIGHEST_STABLE=$(uv run --no-sync --with onyx-devtools ods latest-stable-tag) || {
-              echo "::error::Failed to determine highest stable tag via 'ods latest-stable-tag'"
-              exit 1
-            }
-            if [[ "$TAG" == "$HIGHEST_STABLE" ]]; then
-              IS_LATEST=true
-            fi
-          fi
-
-          # Build craft-latest backend alongside the regular latest.
-          if [[ "$IS_LATEST" == "true" ]]; then
-            BUILD_BACKEND_CRAFT=true
-          fi
-
           # Determine if this is a production tag
           # Production tags are: version tags (v1.2.3*) or nightly tags
           if [[ "$IS_VERSION_TAG" == "true" ]] || [[ "$IS_NIGHTLY" == "true" ]]; then
@@ -145,9 +129,11 @@ jobs:
             echo "build-backend-craft=$BUILD_BACKEND_CRAFT"
             echo "build-model-server=$BUILD_MODEL_SERVER"
             echo "is-cloud-tag=$IS_CLOUD"
+            echo "is-stable=$IS_STABLE"
             echo "is-beta=$IS_BETA"
+            echo "is-stable-standalone=$IS_STABLE_STANDALONE"
             echo "is-beta-standalone=$IS_BETA_STANDALONE"
-            echo "is-latest=$IS_LATEST"
+            echo "is-craft-latest=$IS_CRAFT_LATEST"
             echo "is-test-run=$IS_TEST_RUN"
             echo "sanitized-tag=$SANITIZED_TAG"
             echo "short-sha=$SHORT_SHA"
@@ -165,7 +151,7 @@ jobs:
           fetch-depth: 0
 
       - name: Setup uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           version: "0.9.9"
           # NOTE: This isn't caching much and zizmor suggests this could be poisoned, so disable.
@@ -196,52 +182,8 @@ jobs:
           title: "🚨 Version Tag Check Failed"
           ref-name: ${{ github.ref_name }}
 
-  # Create GitHub release first, before desktop builds start.
-  # This ensures all desktop matrix jobs upload to the same release instead of
-  # racing to create duplicate releases.
-  create-release:
-    needs: determine-builds
-    if: needs.determine-builds.outputs.build-desktop == 'true'
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    permissions:
-      contents: write
-    outputs:
-      release-id: ${{ steps.create-release.outputs.id }}
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Determine release tag
-        id: release-tag
-        env:
-          IS_TEST_RUN: ${{ needs.determine-builds.outputs.is-test-run }}
-          SHORT_SHA: ${{ needs.determine-builds.outputs.short-sha }}
-        run: |
-          if [ "${IS_TEST_RUN}" == "true" ]; then
-            echo "tag=v0.0.0-dev+${SHORT_SHA}" >> "$GITHUB_OUTPUT"
-          else
-            echo "tag=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Create GitHub Release
-        id: create-release
-        uses: softprops/action-gh-release@da05d552573ad5aba039eaac05058a918a7bf631 # ratchet:softprops/action-gh-release@v2
-        with:
-          tag_name: ${{ steps.release-tag.outputs.tag }}
-          name: ${{ steps.release-tag.outputs.tag }}
-          body: "See the assets to download this version and install."
-          draft: true
-          prerelease: false
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
   build-desktop:
-    needs:
-      - determine-builds
-      - create-release
+    needs: determine-builds
     if: needs.determine-builds.outputs.build-desktop == 'true'
     permissions:
       id-token: write
@@ -266,12 +208,12 @@ jobs:
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6.0.2
         with:
-          # NOTE: persist-credentials is needed for tauri-action to upload assets to GitHub releases.
+          # NOTE: persist-credentials is needed for tauri-action to create GitHub releases.
           persist-credentials: true # zizmor: ignore[artipacked]
 
       - name: Configure AWS credentials
         if: startsWith(matrix.platform, 'macos-')
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -411,9 +353,11 @@ jobs:
           APPLE_SIGNING_IDENTITY: ${{ env.CERT_ID }}
           APPLE_TEAM_ID: ${{ env.APPLE_TEAM_ID }}
         with:
-          # Use the release created by the create-release job to avoid race conditions
-          # when multiple matrix jobs try to create/update the same release simultaneously
-          releaseId: ${{ needs.create-release.outputs.release-id }}
+          tagName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseName: ${{ needs.determine-builds.outputs.is-test-run != 'true' && 'v__VERSION__' || format('v0.0.0-dev+{0}', needs.determine-builds.outputs.short-sha) }}
+          releaseBody: "See the assets to download this version and install."
+          releaseDraft: true
+          prerelease: false
           assetNamePattern: "[name]_[arch][ext]"
           args: ${{ matrix.args }}
 
@@ -440,7 +384,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -482,9 +426,8 @@ jobs:
             ONYX_VERSION=${{ github.ref_name }}
             NODE_OPTIONS=--max-old-space-size=8192
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-amd64,mode=max
@@ -514,7 +457,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -556,9 +499,8 @@ jobs:
             ONYX_VERSION=${{ github.ref_name }}
             NODE_OPTIONS=--max-old-space-size=8192
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:web-cache-arm64,mode=max
@@ -583,7 +525,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -614,7 +556,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('web-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta == 'true' && 'beta' || '' }}
 
@@ -653,7 +595,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -704,8 +646,8 @@ jobs:
             NEXT_PUBLIC_INCLUDE_ERROR_POPUP_SUPPORT_LINK=true
             NODE_OPTIONS=--max-old-space-size=8192
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-amd64,mode=max
@@ -735,7 +677,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -786,8 +728,8 @@ jobs:
             NEXT_PUBLIC_INCLUDE_ERROR_POPUP_SUPPORT_LINK=true
             NODE_OPTIONS=--max-old-space-size=8192
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:cloudweb-cache-arm64,mode=max
@@ -812,7 +754,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -879,7 +821,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -920,9 +862,8 @@ jobs:
           build-args: |
             ONYX_VERSION=${{ github.ref_name }}
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-amd64,mode=max
@@ -952,7 +893,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -993,9 +934,8 @@ jobs:
           build-args: |
             ONYX_VERSION=${{ github.ref_name }}
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-cache-arm64,mode=max
@@ -1020,7 +960,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1051,7 +991,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('backend-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable-standalone == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta-standalone == 'true' && 'beta' || '' }}
 
@@ -1090,7 +1030,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1132,8 +1072,8 @@ jobs:
             ONYX_VERSION=${{ github.ref_name }}
             ENABLE_CRAFT=true
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-amd64,mode=max
@@ -1163,7 +1103,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1205,8 +1145,8 @@ jobs:
             ONYX_VERSION=${{ github.ref_name }}
             ENABLE_CRAFT=true
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:backend-craft-cache-arm64,mode=max
@@ -1232,7 +1172,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1302,7 +1242,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1347,9 +1287,8 @@ jobs:
           build-args: |
             ONYX_VERSION=${{ github.ref_name }}
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-amd64,mode=max
@@ -1382,7 +1321,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1427,9 +1366,8 @@ jobs:
           build-args: |
             ONYX_VERSION=${{ github.ref_name }}
           cache-from: |
-            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64
-            type=registry,ref=${{ env.REGISTRY_IMAGE }}:edge
             type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
+            type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64
           cache-to: |
             type=inline
             type=registry,ref=${{ env.RUNS_ON_ECR_CACHE }}:model-server-cache-arm64,mode=max
@@ -1456,7 +1394,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1487,7 +1425,7 @@ jobs:
             latest=false
           tags: |
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run == 'true' && format('model-server-{0}', needs.determine-builds.outputs.sanitized-tag) || github.ref_name }}
-            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-latest == 'true' && 'latest' || '' }}
+            type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-stable-standalone == 'true' && 'latest' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && env.EDGE_TAG == 'true' && 'edge' || '' }}
             type=raw,value=${{ needs.determine-builds.outputs.is-test-run != 'true' && needs.determine-builds.outputs.is-beta-standalone == 'true' && 'beta' || '' }}
 
@@ -1521,7 +1459,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1576,7 +1514,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1636,7 +1574,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -1693,7 +1631,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2

.github/workflows/helm-chart-releases.yml

@@ -33,7 +33,7 @@ jobs:
           helm repo add cloudnative-pg https://cloudnative-pg.github.io/charts
           helm repo add ot-container-kit https://ot-container-kit.github.io/helm-charts
           helm repo add minio https://charts.min.io/
-          helm repo add code-interpreter https://onyx-dot-app.github.io/python-sandbox/
+          helm repo add code-interpreter https://onyx-dot-app.github.io/code-interpreter/
           helm repo update
 
       - name: Build chart dependencies

.github/workflows/nightly-llm-provider-chat.yml

@@ -1,51 +0,0 @@
-name: Nightly LLM Provider Chat Tests
-concurrency:
-  group: Nightly-LLM-Provider-Chat-${{ github.workflow }}-${{ github.ref_name }}
-  cancel-in-progress: true
-
-on:
-  schedule:
-    # Runs daily at 10:30 UTC (2:30 AM PST / 3:30 AM PDT)
-    - cron: "30 10 * * *"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-jobs:
-  provider-chat-test:
-    uses: ./.github/workflows/reusable-nightly-llm-provider-chat.yml
-    secrets:
-      AWS_OIDC_ROLE_ARN: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-    permissions:
-      contents: read
-      id-token: write
-    with:
-      openai_models: ${{ vars.NIGHTLY_LLM_OPENAI_MODELS }}
-      anthropic_models: ${{ vars.NIGHTLY_LLM_ANTHROPIC_MODELS }}
-      bedrock_models: ${{ vars.NIGHTLY_LLM_BEDROCK_MODELS }}
-      vertex_ai_models: ${{ vars.NIGHTLY_LLM_VERTEX_AI_MODELS }}
-      azure_models: ${{ vars.NIGHTLY_LLM_AZURE_MODELS }}
-      azure_api_base: ${{ vars.NIGHTLY_LLM_AZURE_API_BASE }}
-      ollama_models: ${{ vars.NIGHTLY_LLM_OLLAMA_MODELS }}
-      openrouter_models: ${{ vars.NIGHTLY_LLM_OPENROUTER_MODELS }}
-      strict: true
-
-  notify-slack-on-failure:
-    needs: [provider-chat-test]
-    if: failure() && github.event_name == 'schedule'
-    runs-on: ubuntu-slim
-    timeout-minutes: 5
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Send Slack notification
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.SLACK_WEBHOOK }}
-          failed-jobs: provider-chat-test
-          title: "🚨 Scheduled LLM Provider Chat Tests failed!"
-          ref-name: ${{ github.ref_name }}

.github/workflows/nightly-scan-licenses.yml

@@ -0,0 +1,151 @@
+# Scan for problematic software licenses
+
+# trivy has their own rate limiting issues causing this action to flake
+# we worked around it by hardcoding to different db repos in env
+# can re-enable when they figure it out
+# https://github.com/aquasecurity/trivy/discussions/7538
+# https://github.com/aquasecurity/trivy-action/issues/389
+
+name: 'Nightly - Scan licenses'
+on:
+#   schedule:
+#     - cron: '0 14 * * *'  # Runs every day at 6 AM PST / 7 AM PDT / 2 PM UTC
+  workflow_dispatch:  # Allows manual triggering
+
+permissions:
+  actions: read
+  contents: read
+
+jobs:
+  scan-licenses:
+    # See https://runs-on.com/runners/linux/
+    runs-on: [runs-on,runner=2cpu-linux-x64,"run-id=${{ github.run_id }}-scan-licenses"]
+    timeout-minutes: 45
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # ratchet:actions/setup-python@v6
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+          cache-dependency-path: |
+            backend/requirements/default.txt
+            backend/requirements/dev.txt
+            backend/requirements/model_server.txt
+
+      - name: Get explicit and transitive dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install --retries 5 --timeout 30 -r backend/requirements/default.txt
+          pip install --retries 5 --timeout 30 -r backend/requirements/dev.txt
+          pip install --retries 5 --timeout 30 -r backend/requirements/model_server.txt
+          pip freeze > requirements-all.txt
+
+      - name: Check python
+        id: license_check_report
+        uses: pilosus/action-pip-license-checker@e909b0226ff49d3235c99c4585bc617f49fff16a # ratchet:pilosus/action-pip-license-checker@v3
+        with:
+          requirements: 'requirements-all.txt'
+          fail: 'Copyleft'
+          exclude: '(?i)^(pylint|aio[-_]*).*'
+
+      - name: Print report
+        if: always()
+        env:
+          REPORT: ${{ steps.license_check_report.outputs.report }}
+        run: echo "$REPORT"
+
+      - name: Install npm dependencies
+        working-directory: ./web
+        run: npm ci
+
+        # be careful enabling the sarif and upload as it may spam the security tab
+        # with a huge amount of items. Work out the issues before enabling upload.
+#       - name: Run Trivy vulnerability scanner in repo mode
+#         if: always()
+#         uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # ratchet:aquasecurity/trivy-action@0.33.1
+#         with:
+#           scan-type: fs
+#           scan-ref: .
+#           scanners: license
+#           format: table
+#           severity: HIGH,CRITICAL
+# #           format: sarif
+# #           output: trivy-results.sarif
+#
+# #       - name: Upload Trivy scan results to GitHub Security tab
+# #         uses: github/codeql-action/upload-sarif@v3
+# #         with:
+# #           sarif_file: trivy-results.sarif
+
+  scan-trivy:
+    # See https://runs-on.com/runners/linux/
+    runs-on: [runs-on,runner=2cpu-linux-x64,"run-id=${{ github.run_id }}-scan-trivy"]
+    timeout-minutes: 45
+
+    steps:
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # ratchet:docker/setup-buildx-action@v3
+
+    - name: Login to Docker Hub
+      uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
+      with:
+        username: ${{ secrets.DOCKER_USERNAME }}
+        password: ${{ secrets.DOCKER_TOKEN }}
+
+    # Backend
+    - name: Pull backend docker image
+      run: docker pull onyxdotapp/onyx-backend:latest
+
+    - name: Run Trivy vulnerability scanner on backend
+      uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # ratchet:aquasecurity/trivy-action@0.33.1
+      env:
+        TRIVY_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-db:2'
+        TRIVY_JAVA_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-java-db:1'
+      with:
+        image-ref: onyxdotapp/onyx-backend:latest
+        scanners: license
+        severity: HIGH,CRITICAL
+        vuln-type: library
+        exit-code: 0  # Set to 1 if we want a failed scan to fail the workflow
+
+    # Web server
+    - name: Pull web server docker image
+      run: docker pull onyxdotapp/onyx-web-server:latest
+
+    - name: Run Trivy vulnerability scanner on web server
+      uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # ratchet:aquasecurity/trivy-action@0.33.1
+      env:
+        TRIVY_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-db:2'
+        TRIVY_JAVA_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-java-db:1'
+      with:
+        image-ref: onyxdotapp/onyx-web-server:latest
+        scanners: license
+        severity: HIGH,CRITICAL
+        vuln-type: library
+        exit-code: 0
+
+    # Model server
+    - name: Pull model server docker image
+      run: docker pull onyxdotapp/onyx-model-server:latest
+
+    - name: Run Trivy vulnerability scanner
+      uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # ratchet:aquasecurity/trivy-action@0.33.1
+      env:
+        TRIVY_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-db:2'
+        TRIVY_JAVA_DB_REPOSITORY: 'public.ecr.aws/aquasecurity/trivy-java-db:1'
+      with:
+        image-ref: onyxdotapp/onyx-model-server:latest
+        scanners: license
+        severity: HIGH,CRITICAL
+        vuln-type: library
+        exit-code: 0

.github/workflows/post-merge-beta-cherry-pick.yml

@@ -1,294 +0,0 @@
-name: Post-Merge Beta Cherry-Pick
-
-on:
-  pull_request_target:
-    types:
-      - closed
-
-# SECURITY NOTE:
-# This workflow intentionally uses pull_request_target so post-merge automation can
-# use base-repo credentials. Do not checkout PR head refs in this workflow
-# (e.g. github.event.pull_request.head.sha). Only trusted base refs are allowed.
-permissions:
-  contents: read
-
-jobs:
-  resolve-cherry-pick-request:
-    if: >-
-      github.event.pull_request.merged == true
-      && github.event.pull_request.base.ref == 'main'
-      && github.event.pull_request.head.repo.full_name == github.repository
-    outputs:
-      should_cherrypick: ${{ steps.gate.outputs.should_cherrypick }}
-      pr_number: ${{ steps.gate.outputs.pr_number }}
-      merge_commit_sha: ${{ steps.gate.outputs.merge_commit_sha }}
-      merged_by: ${{ steps.gate.outputs.merged_by }}
-      gate_error: ${{ steps.gate.outputs.gate_error }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - name: Resolve merged PR and checkbox state
-        id: gate
-        env:
-          GH_TOKEN: ${{ github.token }}
-          PR_NUMBER: ${{ github.event.pull_request.number }}
-          # SECURITY: keep PR body in env/plain-text handling; avoid directly
-          # inlining github.event.pull_request.body into shell commands.
-          PR_BODY: ${{ github.event.pull_request.body }}
-          MERGE_COMMIT_SHA: ${{ github.event.pull_request.merge_commit_sha }}
-          MERGED_BY: ${{ github.event.pull_request.merged_by.login }}
-          # Explicit merger allowlist used because pull_request_target runs with
-          # the default GITHUB_TOKEN, which cannot reliably read org/team
-          # membership for this repository context.
-          ALLOWED_MERGERS: |
-            acaprau
-            bo-onyx
-            danelegend
-            duo-onyx
-            evan-onyx
-            jessicasingh7
-            jmelahman
-            joachim-danswer
-            justin-tahara
-            nmgarza5
-            raunakab
-            rohoswagger
-            subash-mohan
-            trial2onyx
-            wenxi-onyx
-            weves
-            yuhongsun96
-        run: |
-          echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
-          echo "merged_by=${MERGED_BY}" >> "$GITHUB_OUTPUT"
-
-          if ! echo "${PR_BODY}" | grep -qiE "\\[x\\][[:space:]]*(\\[[^]]+\\][[:space:]]*)?Please cherry-pick this PR to the latest release version"; then
-            echo "should_cherrypick=false" >> "$GITHUB_OUTPUT"
-            echo "Cherry-pick checkbox not checked for PR #${PR_NUMBER}. Skipping."
-            exit 0
-          fi
-
-          # Keep should_cherrypick output before any possible exit 1 below so
-          # notify-slack can still gate on this output even if this job fails.
-          echo "should_cherrypick=true" >> "$GITHUB_OUTPUT"
-          echo "Cherry-pick checkbox checked for PR #${PR_NUMBER}."
-
-          if [ -z "${MERGE_COMMIT_SHA}" ] || [ "${MERGE_COMMIT_SHA}" = "null" ]; then
-            echo "gate_error=missing-merge-commit-sha" >> "$GITHUB_OUTPUT"
-            echo "::error::PR #${PR_NUMBER} requested cherry-pick, but merge_commit_sha is missing."
-            exit 1
-          fi
-
-          echo "merge_commit_sha=${MERGE_COMMIT_SHA}" >> "$GITHUB_OUTPUT"
-
-          normalized_merged_by="$(printf '%s' "${MERGED_BY}" | tr '[:upper:]' '[:lower:]')"
-          normalized_allowed_mergers="$(printf '%s\n' "${ALLOWED_MERGERS}" | tr '[:upper:]' '[:lower:]')"
-          if ! printf '%s\n' "${normalized_allowed_mergers}" | grep -Fxq "${normalized_merged_by}"; then
-            echo "gate_error=not-allowed-merger" >> "$GITHUB_OUTPUT"
-            echo "::error::${MERGED_BY} is not in the explicit cherry-pick merger allowlist. Failing cherry-pick gate."
-            exit 1
-          fi
-
-          exit 0
-
-  cherry-pick-to-latest-release:
-    needs:
-      - resolve-cherry-pick-request
-    if: needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && needs.resolve-cherry-pick-request.result == 'success'
-    permissions:
-      contents: write
-      pull-requests: write
-    outputs:
-      cherry_pick_pr_url: ${{ steps.run_cherry_pick.outputs.pr_url }}
-      cherry_pick_reason: ${{ steps.run_cherry_pick.outputs.reason }}
-      cherry_pick_details: ${{ steps.run_cherry_pick.outputs.details }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 45
-    steps:
-      - name: Checkout repository
-        # SECURITY: keep checkout pinned to trusted base branch; do not switch to PR head refs.
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          fetch-depth: 0
-          persist-credentials: true
-          ref: main
-
-      - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          enable-cache: false
-          version: "0.9.9"
-
-      - name: Configure git identity
-        run: |
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-
-      - name: Create cherry-pick PR to latest release
-        id: run_cherry_pick
-        env:
-          GH_TOKEN: ${{ github.token }}
-          GITHUB_TOKEN: ${{ github.token }}
-          CHERRY_PICK_ASSIGNEE: ${{ needs.resolve-cherry-pick-request.outputs.merged_by }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
-        run: |
-          output_file="$(mktemp)"
-          set +e
-          uv run --no-sync --with onyx-devtools ods cherry-pick "${MERGE_COMMIT_SHA}" --yes --no-verify 2>&1 | tee "$output_file"
-          pipe_statuses=("${PIPESTATUS[@]}")
-          exit_code="${pipe_statuses[0]}"
-          tee_exit="${pipe_statuses[1]:-0}"
-          set -e
-          if [ "${tee_exit}" -ne 0 ]; then
-            echo "status=failure" >> "$GITHUB_OUTPUT"
-            echo "reason=output-capture-failed" >> "$GITHUB_OUTPUT"
-            echo "::error::tee failed to capture cherry-pick output (exit ${tee_exit}); cannot classify result."
-            exit 1
-          fi
-
-          if [ "${exit_code}" -eq 0 ]; then
-            pr_url="$(sed -n 's/^.*PR created successfully: \(https:\/\/github\.com\/[^[:space:]]\+\/pull\/[0-9]\+\).*$/\1/p' "$output_file" | tail -n 1)"
-            echo "status=success" >> "$GITHUB_OUTPUT"
-            if [ -n "${pr_url}" ]; then
-              echo "pr_url=${pr_url}" >> "$GITHUB_OUTPUT"
-            fi
-            exit 0
-          fi
-
-          echo "status=failure" >> "$GITHUB_OUTPUT"
-
-          reason="command-failed"
-          if grep -qiE "merge conflict during cherry-pick|CONFLICT|could not apply|cherry-pick in progress with staged changes" "$output_file"; then
-            reason="merge-conflict"
-          fi
-          echo "reason=${reason}" >> "$GITHUB_OUTPUT"
-
-          {
-            echo "details<<EOF"
-            tail -n 40 "$output_file"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Mark workflow as failed if cherry-pick failed
-        if: steps.run_cherry_pick.outputs.status == 'failure'
-        env:
-          CHERRY_PICK_REASON: ${{ steps.run_cherry_pick.outputs.reason }}
-        run: |
-          echo "::error::Automated cherry-pick failed (${CHERRY_PICK_REASON})."
-          exit 1
-
-  notify-slack-on-cherry-pick-success:
-    needs:
-      - resolve-cherry-pick-request
-      - cherry-pick-to-latest-release
-    if: needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && needs.resolve-cherry-pick-request.result == 'success' && needs.cherry-pick-to-latest-release.result == 'success'
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Fail if Slack webhook secret is missing
-        env:
-          CHERRY_PICK_PRS_WEBHOOK: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-        run: |
-          if [ -z "${CHERRY_PICK_PRS_WEBHOOK}" ]; then
-            echo "::error::CHERRY_PICK_PRS_WEBHOOK is not configured."
-            exit 1
-          fi
-
-      - name: Build cherry-pick success summary
-        id: success-summary
-        env:
-          SOURCE_PR_NUMBER: ${{ needs.resolve-cherry-pick-request.outputs.pr_number }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
-          CHERRY_PICK_PR_URL: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_pr_url }}
-        run: |
-          source_pr_url="https://github.com/${GITHUB_REPOSITORY}/pull/${SOURCE_PR_NUMBER}"
-          details="*Cherry-pick PR opened successfully.*\\n• source PR: ${source_pr_url}"
-          if [ -n "${CHERRY_PICK_PR_URL}" ]; then
-            details="${details}\\n• cherry-pick PR: ${CHERRY_PICK_PR_URL}"
-          fi
-          if [ -n "${MERGE_COMMIT_SHA}" ]; then
-            details="${details}\\n• merge SHA: ${MERGE_COMMIT_SHA}"
-          fi
-
-          echo "details=${details}" >> "$GITHUB_OUTPUT"
-
-      - name: Notify #cherry-pick-prs about cherry-pick success
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-          details: ${{ steps.success-summary.outputs.details }}
-          title: "✅ Automated Cherry-Pick PR Opened"
-          ref-name: ${{ github.event.pull_request.base.ref }}
-
-  notify-slack-on-cherry-pick-failure:
-    needs:
-      - resolve-cherry-pick-request
-      - cherry-pick-to-latest-release
-    if: always() && needs.resolve-cherry-pick-request.outputs.should_cherrypick == 'true' && (needs.resolve-cherry-pick-request.result == 'failure' || needs.cherry-pick-to-latest-release.result == 'failure')
-    runs-on: ubuntu-slim
-    timeout-minutes: 10
-    steps:
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Fail if Slack webhook secret is missing
-        env:
-          CHERRY_PICK_PRS_WEBHOOK: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-        run: |
-          if [ -z "${CHERRY_PICK_PRS_WEBHOOK}" ]; then
-            echo "::error::CHERRY_PICK_PRS_WEBHOOK is not configured."
-            exit 1
-          fi
-
-      - name: Build cherry-pick failure summary
-        id: failure-summary
-        env:
-          SOURCE_PR_NUMBER: ${{ needs.resolve-cherry-pick-request.outputs.pr_number }}
-          MERGE_COMMIT_SHA: ${{ needs.resolve-cherry-pick-request.outputs.merge_commit_sha }}
-          GATE_ERROR: ${{ needs.resolve-cherry-pick-request.outputs.gate_error }}
-          CHERRY_PICK_REASON: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_reason }}
-          CHERRY_PICK_DETAILS: ${{ needs.cherry-pick-to-latest-release.outputs.cherry_pick_details }}
-        run: |
-          source_pr_url="https://github.com/${GITHUB_REPOSITORY}/pull/${SOURCE_PR_NUMBER}"
-
-          reason_text="cherry-pick command failed"
-          if [ "${GATE_ERROR}" = "missing-merge-commit-sha" ]; then
-            reason_text="requested cherry-pick but merge commit SHA was missing"
-          elif [ "${GATE_ERROR}" = "not-allowed-merger" ]; then
-            reason_text="merger is not in the explicit cherry-pick allowlist"
-          elif [ "${CHERRY_PICK_REASON}" = "output-capture-failed" ]; then
-            reason_text="failed to capture cherry-pick output for classification"
-          elif [ "${CHERRY_PICK_REASON}" = "merge-conflict" ]; then
-            reason_text="merge conflict during cherry-pick"
-          fi
-
-          details_excerpt="$(printf '%s' "${CHERRY_PICK_DETAILS}" | tail -n 8 | tr '\n' ' ' | sed "s/[[:space:]]\\+/ /g" | sed "s/\"/'/g" | cut -c1-350)"
-          if [ -n "${GATE_ERROR}" ]; then
-            failed_job_label="resolve-cherry-pick-request"
-          else
-            failed_job_label="cherry-pick-to-latest-release"
-          fi
-          failed_jobs="• ${failed_job_label}\\n• source PR: ${source_pr_url}\\n• reason: ${reason_text}"
-          if [ -n "${MERGE_COMMIT_SHA}" ]; then
-            failed_jobs="${failed_jobs}\\n• merge SHA: ${MERGE_COMMIT_SHA}"
-          fi
-          if [ -n "${details_excerpt}" ]; then
-            failed_jobs="${failed_jobs}\\n• excerpt: ${details_excerpt}"
-          fi
-
-          echo "jobs=${failed_jobs}" >> "$GITHUB_OUTPUT"
-
-      - name: Notify #cherry-pick-prs about cherry-pick failure
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.CHERRY_PICK_PRS_WEBHOOK }}
-          details: ${{ steps.failure-summary.outputs.jobs }}
-          title: "🚨 Automated Cherry-Pick Failed"
-          ref-name: ${{ github.event.pull_request.base.ref }}

.github/workflows/pr-beta-cherrypick-check.yml

@@ -0,0 +1,28 @@
+name: Require beta cherry-pick consideration
+concurrency:
+  group: Require-Beta-Cherrypick-Consideration-${{ github.workflow }}-${{ github.head_ref || github.event.workflow_run.head_branch || github.run_id }}
+  cancel-in-progress: true
+
+on:
+  pull_request:
+    types: [opened, edited, reopened, synchronize]
+
+permissions:
+  contents: read
+
+jobs:
+  beta-cherrypick-check:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Check PR body for beta cherry-pick consideration
+        env:
+          PR_BODY: ${{ github.event.pull_request.body }}
+        run: |
+          if echo "$PR_BODY" | grep -qiE "\\[x\\][[:space:]]*\\[Required\\][[:space:]]*I have considered whether this PR needs to be cherry[- ]picked to the latest beta branch"; then
+            echo "Cherry-pick consideration box is checked. Check passed."
+            exit 0
+          fi
+
+          echo "::error::Please check the 'I have considered whether this PR needs to be cherry-picked to the latest beta branch' box in the PR description."
+          exit 1

.github/workflows/pr-desktop-build.yml

@@ -57,7 +57,7 @@ jobs:
           cache-dependency-path: ./desktop/package-lock.json
 
       - name: Setup Rust
-        uses: dtolnay/rust-toolchain@efa25f7f19611383d5b0ccf2d1c8914531636bf9
+        uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7
         with:
           toolchain: stable
           targets: ${{ matrix.target }}

.github/workflows/pr-external-dependency-unit-tests.yml

@@ -45,6 +45,9 @@ env:
   # TODO: debug why this is failing and enable
   CODE_INTERPRETER_BASE_URL: http://localhost:8000
 
+  # OpenSearch
+  OPENSEARCH_ADMIN_PASSWORD: "StrongPassword123!"
+
 jobs:
   discover-test-dirs:
     # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
@@ -115,9 +118,9 @@ jobs:
       - name: Create .env file for Docker Compose
         run: |
           cat <<EOF > deployment/docker_compose/.env
-          COMPOSE_PROFILES=s3-filestore,opensearch-enabled
+          COMPOSE_PROFILES=s3-filestore
+          CODE_INTERPRETER_BETA_ENABLED=true
           DISABLE_TELEMETRY=true
-          OPENSEARCH_FOR_ONYX_ENABLED=true
           EOF
 
       - name: Set up Standard Dependencies
@@ -126,6 +129,7 @@ jobs:
           docker compose \
             -f docker-compose.yml \
             -f docker-compose.dev.yml \
+            -f docker-compose.opensearch.yml \
             up -d \
             minio \
             relational_db \
@@ -160,7 +164,7 @@ jobs:
           cd deployment/docker_compose
 
           # Get list of running containers
-          containers=$(docker compose -f docker-compose.yml -f docker-compose.dev.yml ps -q)
+          containers=$(docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.opensearch.yml ps -q)
 
           # Collect logs from each container
           for container in $containers; do

.github/workflows/pr-golang-tests.yml

@@ -1,56 +0,0 @@
-name: Golang Tests
-concurrency:
-  group: Golang-Tests-${{ github.workflow }}-${{ github.head_ref || github.event.workflow_run.head_branch || github.run_id }}
-  cancel-in-progress: true
-
-on:
-  merge_group:
-  pull_request:
-    branches:
-      - main
-      - "release/**"
-  push:
-    tags:
-      - "v*.*.*"
-
-permissions: {}
-
-env:
-  GO_VERSION: "1.26"
-
-jobs:
-  detect-modules:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      modules: ${{ steps.set-modules.outputs.modules }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
-        with:
-          persist-credentials: false
-      - id: set-modules
-        run: echo "modules=$(find . -name 'go.mod' -exec dirname {} \; | jq -Rc '[.,inputs]')" >> "$GITHUB_OUTPUT"
-
-  golang:
-    needs: detect-modules
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        modules: ${{ fromJSON(needs.detect-modules.outputs.modules) }}
-    steps:
-      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # zizmor: ignore[cache-poisoning]
-        with:
-          go-version: ${{ env.GO_VERSION }}
-          cache-dependency-path: "**/go.sum"
-
-      - run: go mod tidy
-        working-directory: ${{ matrix.modules }}
-      - run: git diff --exit-code go.mod go.sum
-        working-directory: ${{ matrix.modules }}
-
-      - run: go test ./...
-        working-directory: ${{ matrix.modules }}

.github/workflows/pr-helm-chart-testing.yml

@@ -41,7 +41,8 @@ jobs:
           version: v3.19.0
 
       - name: Set up chart-testing
-        uses: helm/chart-testing-action@b5eebdd9998021f29756c53432f48dab66394810
+        # NOTE: This is Jamison's patch from https://github.com/helm/chart-testing-action/pull/194
+        uses: helm/chart-testing-action@8958a6ac472cbd8ee9a8fbb6f1acbc1b0e966e44 # zizmor: ignore[impostor-commit]
         with:
           uv_version: "0.9.9"
 
@@ -71,7 +72,7 @@ jobs:
 
       - name: Create kind cluster
         if: steps.list-changed.outputs.changed == 'true'
-        uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # ratchet:helm/kind-action@v1.14.0
+        uses: helm/kind-action@92086f6be054225fa813e0a4b13787fc9088faab # ratchet:helm/kind-action@v1.13.0
 
       - name: Pre-install cluster status check
         if: steps.list-changed.outputs.changed == 'true'
@@ -91,7 +92,7 @@ jobs:
           helm repo add cloudnative-pg https://cloudnative-pg.github.io/charts
           helm repo add ot-container-kit https://ot-container-kit.github.io/helm-charts
           helm repo add minio https://charts.min.io/
-          helm repo add code-interpreter https://onyx-dot-app.github.io/python-sandbox/
+          helm repo add code-interpreter https://onyx-dot-app.github.io/code-interpreter/
           helm repo update
 
       - name: Install Redis operator
@@ -133,7 +134,7 @@ jobs:
           echo "=== Validating chart dependencies ==="
           cd deployment/helm/charts/onyx
           helm dependency update
-          helm lint . --set auth.userauth.values.user_auth_secret=placeholder
+          helm lint .
 
       - name: Run chart-testing (install) with enhanced monitoring
         timeout-minutes: 25
@@ -194,7 +195,6 @@ jobs:
               --set=vespa.enabled=false \
               --set=opensearch.enabled=true \
               --set=auth.opensearch.enabled=true \
-              --set=auth.userauth.values.user_auth_secret=test-secret \
               --set=slackbot.enabled=false \
               --set=postgresql.enabled=true \
               --set=postgresql.cluster.storage.storageClass=standard \
@@ -231,10 +231,6 @@ jobs:
         if: steps.list-changed.outputs.changed == 'true'
         run: |
           echo "=== Post-install verification ==="
-          if ! kubectl cluster-info >/dev/null 2>&1; then
-            echo "ERROR: Kubernetes cluster is not reachable after install"
-            exit 1
-          fi
           kubectl get pods --all-namespaces
           kubectl get services --all-namespaces
           # Only show issues if they exist
@@ -244,10 +240,6 @@ jobs:
         if: failure() && steps.list-changed.outputs.changed == 'true'
         run: |
           echo "=== Cleanup on failure ==="
-          if ! kubectl cluster-info >/dev/null 2>&1; then
-            echo "Skipping failure cleanup: Kubernetes cluster is not reachable"
-            exit 0
-          fi
           echo "=== Final cluster state ==="
           kubectl get pods --all-namespaces
           kubectl get events --all-namespaces --sort-by=.lastTimestamp | tail -10

.github/workflows/pr-integration-tests.yml

@@ -20,7 +20,6 @@ env:
   # Test Environment Variables
   OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
   SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
-  SLACK_BOT_TOKEN_TEST_SPACE: ${{ secrets.SLACK_BOT_TOKEN_TEST_SPACE }}
   CONFLUENCE_TEST_SPACE_URL: ${{ vars.CONFLUENCE_TEST_SPACE_URL }}
   CONFLUENCE_USER_NAME: ${{ vars.CONFLUENCE_USER_NAME }}
   CONFLUENCE_ACCESS_TOKEN: ${{ secrets.CONFLUENCE_ACCESS_TOKEN }}
@@ -316,7 +315,6 @@ jobs:
           # Base config shared by both editions
           cat <<EOF > deployment/docker_compose/.env
           COMPOSE_PROFILES=s3-filestore
-          OPENSEARCH_FOR_ONYX_ENABLED=false
           AUTH_TYPE=basic
           POSTGRES_POOL_PRE_PING=true
           POSTGRES_USE_NULL_POOL=true
@@ -336,6 +334,7 @@ jobs:
           # TODO(Nik): https://linear.app/onyx-app/issue/ENG-1/update-test-infra-to-use-test-license
           LICENSE_ENFORCEMENT_ENABLED=false
           CHECK_TTL_MANAGEMENT_TASK_FREQUENCY_IN_HOURS=0.001
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=false
           EOF
           fi
 
@@ -419,13 +418,11 @@ jobs:
               -e POSTGRES_POOL_PRE_PING=true \
               -e POSTGRES_USE_NULL_POOL=true \
               -e VESPA_HOST=index \
-              -e ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false \
               -e REDIS_HOST=cache \
               -e API_SERVER_HOST=api_server \
               -e OPENAI_API_KEY=${OPENAI_API_KEY} \
               -e EXA_API_KEY=${EXA_API_KEY} \
               -e SLACK_BOT_TOKEN=${SLACK_BOT_TOKEN} \
-              -e SLACK_BOT_TOKEN_TEST_SPACE=${SLACK_BOT_TOKEN_TEST_SPACE} \
               -e CONFLUENCE_TEST_SPACE_URL=${CONFLUENCE_TEST_SPACE_URL} \
               -e CONFLUENCE_USER_NAME=${CONFLUENCE_USER_NAME} \
               -e CONFLUENCE_ACCESS_TOKEN=${CONFLUENCE_ACCESS_TOKEN} \
@@ -446,7 +443,6 @@ jobs:
               -e TEST_WEB_HOSTNAME=test-runner \
               -e MOCK_CONNECTOR_SERVER_HOST=mock_connector_server \
               -e MOCK_CONNECTOR_SERVER_PORT=8001 \
-              -e ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=${{ matrix.edition == 'ee' && 'true' || 'false' }} \
               ${{ env.RUNS_ON_ECR_CACHE }}:integration-test-${{ github.run_id }} \
               /app/tests/integration/${{ matrix.test-dir.path }}
 
@@ -472,13 +468,13 @@ jobs:
           path: ${{ github.workspace }}/docker-compose.log
       # ------------------------------------------------------------
 
-  onyx-lite-tests:
+  no-vectordb-tests:
     needs: [build-backend-image, build-integration-image]
     runs-on:
       [
         runs-on,
         runner=4cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-onyx-lite-tests",
+        "run-id=${{ github.run_id }}-no-vectordb-tests",
         "extras=ecr-cache",
       ]
     timeout-minutes: 45
@@ -496,12 +492,13 @@ jobs:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_TOKEN }}
 
-      - name: Create .env file for Onyx Lite Docker Compose
+      - name: Create .env file for no-vectordb Docker Compose
         env:
           ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
           RUN_ID: ${{ github.run_id }}
         run: |
           cat <<EOF > deployment/docker_compose/.env
+          COMPOSE_PROFILES=s3-filestore
           ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
           LICENSE_ENFORCEMENT_ENABLED=false
           AUTH_TYPE=basic
@@ -509,23 +506,28 @@ jobs:
           POSTGRES_USE_NULL_POOL=true
           REQUIRE_EMAIL_VERIFICATION=false
           DISABLE_TELEMETRY=true
+          DISABLE_VECTOR_DB=true
           ONYX_BACKEND_IMAGE=${ECR_CACHE}:integration-test-backend-test-${RUN_ID}
           INTEGRATION_TESTS_MODE=true
+          USE_LIGHTWEIGHT_BACKGROUND_WORKER=true
           EOF
 
-      # Start only the services needed for Onyx Lite (Postgres + API server)
-      - name: Start Docker containers (onyx-lite)
+      # Start only the services needed for no-vectordb mode (no Vespa, no model servers)
+      - name: Start Docker containers (no-vectordb)
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up \
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml up \
             relational_db \
+            cache \
+            minio \
             api_server \
+            background \
             -d
-        id: start_docker_onyx_lite
+        id: start_docker_no_vectordb
 
       - name: Wait for services to be ready
         run: |
-          echo "Starting wait-for-service script (onyx-lite)..."
+          echo "Starting wait-for-service script (no-vectordb)..."
           start_time=$(date +%s)
           timeout=300
           while true; do
@@ -547,14 +549,14 @@ jobs:
             sleep 5
           done
 
-      - name: Run Onyx Lite Integration Tests
+      - name: Run No-VectorDB Integration Tests
         uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # ratchet:nick-fields/retry@v3
         with:
           timeout_minutes: 20
           max_attempts: 3
           retry_wait_seconds: 10
           command: |
-            echo "Running onyx-lite integration tests..."
+            echo "Running no-vectordb integration tests..."
             docker run --rm --network onyx_default \
               --name test-runner \
               -e POSTGRES_HOST=relational_db \
@@ -565,38 +567,39 @@ jobs:
               -e DB_READONLY_PASSWORD=password \
               -e POSTGRES_POOL_PRE_PING=true \
               -e POSTGRES_USE_NULL_POOL=true \
+              -e REDIS_HOST=cache \
               -e API_SERVER_HOST=api_server \
               -e OPENAI_API_KEY=${OPENAI_API_KEY} \
               -e TEST_WEB_HOSTNAME=test-runner \
               ${{ env.RUNS_ON_ECR_CACHE }}:integration-test-${{ github.run_id }} \
               /app/tests/integration/tests/no_vectordb
 
-      - name: Dump API server logs (onyx-lite)
+      - name: Dump API server logs (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_onyx_lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color api_server > $GITHUB_WORKSPACE/api_server_no_vectordb.log || true
 
-      - name: Dump all-container logs (onyx-lite)
+      - name: Dump all-container logs (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml \
-            logs --no-color > $GITHUB_WORKSPACE/docker-compose-onyx-lite.log || true
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml \
+            logs --no-color > $GITHUB_WORKSPACE/docker-compose-no-vectordb.log || true
 
-      - name: Upload logs (onyx-lite)
+      - name: Upload logs (no-vectordb)
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
         with:
-          name: docker-all-logs-onyx-lite
-          path: ${{ github.workspace }}/docker-compose-onyx-lite.log
+          name: docker-all-logs-no-vectordb
+          path: ${{ github.workspace }}/docker-compose-no-vectordb.log
 
-      - name: Stop Docker containers (onyx-lite)
+      - name: Stop Docker containers (no-vectordb)
         if: always()
         run: |
           cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml down -v
+          docker compose -f docker-compose.yml -f docker-compose.no-vectordb.yml -f docker-compose.dev.yml down -v
 
   multitenant-tests:
     needs:
@@ -639,7 +642,6 @@ jobs:
           ONYX_BACKEND_IMAGE=${ECR_CACHE}:integration-test-backend-test-${RUN_ID} \
           ONYX_MODEL_SERVER_IMAGE=${ECR_CACHE}:integration-test-model-server-test-${RUN_ID} \
           DEV_MODE=true \
-          OPENSEARCH_FOR_ONYX_ENABLED=false \
           docker compose -f docker-compose.multitenant-dev.yml up \
             relational_db \
             index \
@@ -694,13 +696,11 @@ jobs:
             -e POSTGRES_DB=postgres \
             -e POSTGRES_USE_NULL_POOL=true \
             -e VESPA_HOST=index \
-            -e ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false \
             -e REDIS_HOST=cache \
             -e API_SERVER_HOST=api_server \
             -e OPENAI_API_KEY=${OPENAI_API_KEY} \
             -e EXA_API_KEY=${EXA_API_KEY} \
             -e SLACK_BOT_TOKEN=${SLACK_BOT_TOKEN} \
-            -e SLACK_BOT_TOKEN_TEST_SPACE=${SLACK_BOT_TOKEN_TEST_SPACE} \
             -e TEST_WEB_HOSTNAME=test-runner \
             -e AUTH_TYPE=cloud \
             -e MULTI_TENANT=true \
@@ -740,7 +740,7 @@ jobs:
     # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
     runs-on: ubuntu-slim
     timeout-minutes: 45
-    needs: [integration-tests, onyx-lite-tests, multitenant-tests]
+    needs: [integration-tests, no-vectordb-tests, multitenant-tests]
     if: ${{ always() }}
     steps:
       - name: Check job status

.github/workflows/pr-jest-tests.yml

@@ -31,7 +31,7 @@ jobs:
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
         with:
           node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning] test-only workflow; no deploy artifacts
+          cache: "npm"
           cache-dependency-path: ./web/package-lock.json
 
       - name: Install node dependencies

.github/workflows/pr-playwright-tests.yml

@@ -12,9 +12,6 @@ on:
   push:
     tags:
       - "v*.*.*"
-    # TODO: Remove this if we enable merge-queues for release branches.
-    branches:
-      - "release/**"
 
 permissions:
   contents: read
@@ -25,9 +22,6 @@ env:
   SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
   GEN_AI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
   EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
-  FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}
-  GOOGLE_PSE_API_KEY: ${{ secrets.GOOGLE_PSE_API_KEY }}
-  GOOGLE_PSE_SEARCH_ENGINE_ID: ${{ secrets.GOOGLE_PSE_SEARCH_ENGINE_ID }}
 
   # for federated slack tests
   SLACK_CLIENT_ID: ${{ secrets.SLACK_CLIENT_ID }}
@@ -271,11 +265,10 @@ jobs:
           persist-credentials: false
 
       - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
         with:
           node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
+          cache: "npm"
           cache-dependency-path: ./web/package-lock.json
 
       - name: Install node dependencies
@@ -283,7 +276,6 @@ jobs:
         run: npm ci
 
       - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
         uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
         with:
           path: ~/.cache/ms-playwright
@@ -308,7 +300,6 @@ jobs:
           # TODO(Nik): https://linear.app/onyx-app/issue/ENG-1/update-test-infra-to-use-test-license
           LICENSE_ENFORCEMENT_ENABLED=false
           AUTH_TYPE=basic
-          INTEGRATION_TESTS_MODE=true
           GEN_AI_API_KEY=${OPENAI_API_KEY_VALUE}
           EXA_API_KEY=${EXA_API_KEY_VALUE}
           REQUIRE_EMAIL_VERIFICATION=false
@@ -464,14 +455,14 @@ jobs:
       # --- Visual Regression Diff ---
       - name: Configure AWS credentials
         if: always()
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
 
       - name: Install the latest version of uv
         if: always()
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"
@@ -595,122 +586,17 @@ jobs:
           name: docker-logs-${{ matrix.project }}-${{ github.run_id }}
           path: ${{ github.workspace }}/docker-compose.log
 
-  playwright-tests-lite:
-    needs: [build-web-image, build-backend-image]
-    name: Playwright Tests (lite)
-    runs-on:
-      - runs-on
-      - runner=4cpu-linux-arm64
-      - "run-id=${{ github.run_id }}-playwright-tests-lite"
-      - "extras=ecr-cache"
-    timeout-minutes: 30
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Setup node
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
-        with:
-          node-version: 22
-          cache: "npm" # zizmor: ignore[cache-poisoning]
-          cache-dependency-path: ./web/package-lock.json
-
-      - name: Install node dependencies
-        working-directory: ./web
-        run: npm ci
-
-      - name: Cache playwright cache
-        # zizmor: ignore[cache-poisoning] ephemeral runners; no release artifacts
-        uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
-        with:
-          path: ~/.cache/ms-playwright
-          key: ${{ runner.os }}-playwright-npm-${{ hashFiles('web/package-lock.json') }}
-          restore-keys: |
-            ${{ runner.os }}-playwright-npm-
-
-      - name: Install playwright browsers
-        working-directory: ./web
-        run: npx playwright install --with-deps
-
-      - name: Create .env file for Docker Compose
-        env:
-          OPENAI_API_KEY_VALUE: ${{ env.OPENAI_API_KEY }}
-          ECR_CACHE: ${{ env.RUNS_ON_ECR_CACHE }}
-          RUN_ID: ${{ github.run_id }}
-        run: |
-          cat <<EOF > deployment/docker_compose/.env
-          ENABLE_PAID_ENTERPRISE_EDITION_FEATURES=true
-          LICENSE_ENFORCEMENT_ENABLED=false
-          AUTH_TYPE=basic
-          INTEGRATION_TESTS_MODE=true
-          GEN_AI_API_KEY=${OPENAI_API_KEY_VALUE}
-          MOCK_LLM_RESPONSE=true
-          REQUIRE_EMAIL_VERIFICATION=false
-          DISABLE_TELEMETRY=true
-          ONYX_BACKEND_IMAGE=${ECR_CACHE}:playwright-test-backend-${RUN_ID}
-          ONYX_WEB_SERVER_IMAGE=${ECR_CACHE}:playwright-test-web-${RUN_ID}
-          EOF
-
-      # needed for pulling external images otherwise, we hit the "Unauthenticated users" limit
-      # https://docs.docker.com/docker-hub/usage/
-      - name: Login to Docker Hub
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # ratchet:docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_TOKEN }}
-
-      - name: Start Docker containers (lite)
-        run: |
-          cd deployment/docker_compose
-          docker compose -f docker-compose.yml -f docker-compose.onyx-lite.yml -f docker-compose.dev.yml up -d
-        id: start_docker
-
-      - name: Run Playwright tests (lite)
-        working-directory: ./web
-        run: npx playwright test --project lite
-
-      - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        if: always()
-        with:
-          name: playwright-test-results-lite-${{ github.run_id }}
-          path: ./web/output/playwright/
-          retention-days: 30
-
-      - name: Save Docker logs
-        if: success() || failure()
-        env:
-          WORKSPACE: ${{ github.workspace }}
-        run: |
-          cd deployment/docker_compose
-          docker compose logs > docker-compose.log
-          mv docker-compose.log ${WORKSPACE}/docker-compose.log
-
-      - name: Upload logs
-        if: success() || failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        with:
-          name: docker-logs-lite-${{ github.run_id }}
-          path: ${{ github.workspace }}/docker-compose.log
-
   # Post a single combined visual regression comment after all matrix jobs finish
   visual-regression-comment:
     needs: [playwright-tests]
-    if: >-
-      always() &&
-      github.event_name == 'pull_request' &&
-      needs.playwright-tests.result != 'cancelled'
+    if: always() && github.event_name == 'pull_request'
     runs-on: ubuntu-slim
     timeout-minutes: 5
     permissions:
       pull-requests: write
     steps:
       - name: Download visual diff summaries
-        uses: actions/download-artifact@70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3
+        uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e # ratchet:actions/download-artifact@v4
         with:
           pattern: screenshot-diff-summary-*
           path: summaries/
@@ -793,7 +679,7 @@ jobs:
     # NOTE: Github-hosted runners have about 20s faster queue times and are preferred here.
     runs-on: ubuntu-slim
     timeout-minutes: 45
-    needs: [playwright-tests, playwright-tests-lite]
+    needs: [playwright-tests]
     if: ${{ always() }}
     steps:
       - name: Check job status

.github/workflows/pr-python-checks.yml

@@ -8,7 +8,7 @@ on:
   pull_request:
     branches:
       - main
-      - "release/**"
+      - 'release/**'
   push:
     tags:
       - "v*.*.*"
@@ -21,13 +21,7 @@ jobs:
     # See https://runs-on.com/runners/linux/
     # Note: Mypy seems quite optimized for x64 compared to arm64.
     # Similarly, mypy is single-threaded and incremental, so 2cpu is sufficient.
-    runs-on:
-      [
-        runs-on,
-        runner=2cpu-linux-x64,
-        "run-id=${{ github.run_id }}-mypy-check",
-        "extras=s3-cache",
-      ]
+    runs-on: [runs-on, runner=2cpu-linux-x64, "run-id=${{ github.run_id }}-mypy-check", "extras=s3-cache"]
     timeout-minutes: 45
 
     steps:
@@ -58,14 +52,21 @@ jobs:
         if: ${{ vars.DISABLE_MYPY_CACHE != 'true' }}
         uses: runs-on/cache@50350ad4242587b6c8c2baa2e740b1bc11285ff4 # ratchet:runs-on/cache@v4
         with:
-          path: .mypy_cache
-          key: mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-${{ hashFiles('**/*.py', '**/*.pyi', 'pyproject.toml') }}
+          path: backend/.mypy_cache
+          key: mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-${{ hashFiles('**/*.py', '**/*.pyi', 'backend/pyproject.toml') }}
           restore-keys: |
             mypy-${{ runner.os }}-${{ github.base_ref || github.event.merge_group.base_ref || 'main' }}-
             mypy-${{ runner.os }}-
 
       - name: Run MyPy
+        working-directory: ./backend
         env:
           MYPY_FORCE_COLOR: 1
           TERM: xterm-256color
         run: mypy .
+
+      - name: Run MyPy (tools/)
+        env:
+          MYPY_FORCE_COLOR: 1
+          TERM: xterm-256color
+        run: mypy tools/

.github/workflows/pr-python-connector-tests.yml

@@ -89,10 +89,6 @@ env:
   SHAREPOINT_CLIENT_SECRET: ${{ secrets.SHAREPOINT_CLIENT_SECRET }}
   SHAREPOINT_CLIENT_DIRECTORY_ID: ${{ vars.SHAREPOINT_CLIENT_DIRECTORY_ID }}
   SHAREPOINT_SITE: ${{ vars.SHAREPOINT_SITE }}
-  PERM_SYNC_SHAREPOINT_CLIENT_ID: ${{ secrets.PERM_SYNC_SHAREPOINT_CLIENT_ID }}
-  PERM_SYNC_SHAREPOINT_PRIVATE_KEY: ${{ secrets.PERM_SYNC_SHAREPOINT_PRIVATE_KEY }}
-  PERM_SYNC_SHAREPOINT_CERTIFICATE_PASSWORD: ${{ secrets.PERM_SYNC_SHAREPOINT_CERTIFICATE_PASSWORD }}
-  PERM_SYNC_SHAREPOINT_DIRECTORY_ID: ${{ secrets.PERM_SYNC_SHAREPOINT_DIRECTORY_ID }}
 
   # Github
   ACCESS_TOKEN_GITHUB: ${{ secrets.ACCESS_TOKEN_GITHUB }}

.github/workflows/pr-quality-checks.yml

@@ -28,7 +28,7 @@ jobs:
         with:
           python-version: "3.11"
       - name: Setup Terraform
-        uses: hashicorp/setup-terraform@5e8dbf3c6d9deaf4193ca7a8fb23f2ac83bb6c85 # ratchet:hashicorp/setup-terraform@v4.0.0
+        uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # ratchet:hashicorp/setup-terraform@v3
       - name: Setup node
         uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v6
         with: # zizmor: ignore[cache-poisoning]
@@ -38,9 +38,9 @@ jobs:
       - name: Install node dependencies
         working-directory: ./web
         run: npm ci
-      - uses: j178/prek-action@0bb87d7f00b0c99306c8bcb8b8beba1eb581c037 # ratchet:j178/prek-action@v1
+      - uses: j178/prek-action@9d6a3097e0c1865ecce00cfb89fe80f2ee91b547 # ratchet:j178/prek-action@v1
         with:
-          prek-version: '0.3.4'
+          prek-version: '0.2.21'
           extra-args: ${{ github.event_name == 'pull_request' && format('--from-ref {0} --to-ref {1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha) || github.event_name == 'merge_group' && format('--from-ref {0} --to-ref {1}', github.event.merge_group.base_sha, github.event.merge_group.head_sha) || github.ref_name == 'main' && '--all-files' || '' }}
       - name: Check Actions
         uses: giner/check-actions@28d366c7cbbe235f9624a88aa31a628167eee28c # ratchet:giner/check-actions@v1.0.1

.github/workflows/release-cli.yml

@@ -1,214 +0,0 @@
-name: Release CLI
-
-on:
-  push:
-    tags:
-      - "cli/v*.*.*"
-
-jobs:
-  pypi:
-    runs-on: ubuntu-latest
-    environment:
-      name: release-cli
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    strategy:
-      matrix:
-        os-arch:
-          - { goos: "linux", goarch: "amd64" }
-          - { goos: "linux", goarch: "arm64" }
-          - { goos: "windows", goarch: "amd64" }
-          - { goos: "windows", goarch: "arm64" }
-          - { goos: "darwin", goarch: "amd64" }
-          - { goos: "darwin", goarch: "arm64" }
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
-        with:
-          enable-cache: false
-          version: "0.9.9"
-      - run: |
-          GOOS="${{ matrix.os-arch.goos }}" \
-          GOARCH="${{ matrix.os-arch.goarch }}" \
-          uv build --wheel
-        working-directory: cli
-      - run: uv publish
-        working-directory: cli
-
-  docker-amd64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-amd64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push AMD64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/amd64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  docker-arm64:
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-arm64
-      - run-id=${{ github.run_id }}-cli-arm64
-      - extras=ecr-cache
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 30
-    outputs:
-      digest: ${{ steps.build.outputs.digest }}
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Build and push ARM64
-        id: build
-        uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # ratchet:docker/build-push-action@v7
-        with:
-          context: ./cli
-          file: ./cli/Dockerfile
-          platforms: linux/arm64
-          cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:latest
-          cache-to: type=inline
-          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true
-
-  merge-docker:
-    needs:
-      - docker-amd64
-      - docker-arm64
-    runs-on:
-      - runs-on
-      - runner=2cpu-linux-x64
-      - run-id=${{ github.run_id }}-cli-merge
-    environment: deploy
-    permissions:
-      id-token: write
-    timeout-minutes: 10
-    env:
-      REGISTRY_IMAGE: onyxdotapp/onyx-cli
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # ratchet:aws-actions/configure-aws-credentials@v6.0.0
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802 # ratchet:aws-actions/aws-secretsmanager-get-secrets@v2.0.10
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, deploy/docker-username
-            DOCKER_TOKEN, deploy/docker-token
-          parse-json-secrets: true
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # ratchet:docker/setup-buildx-action@v4
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # ratchet:docker/login-action@v4
-        with:
-          username: ${{ env.DOCKER_USERNAME }}
-          password: ${{ env.DOCKER_TOKEN }}
-
-      - name: Create and push manifest
-        env:
-          AMD64_DIGEST: ${{ needs.docker-amd64.outputs.digest }}
-          ARM64_DIGEST: ${{ needs.docker-arm64.outputs.digest }}
-          TAG: ${{ github.ref_name }}
-        run: |
-          SANITIZED_TAG="${TAG#cli/}"
-          IMAGES=(
-            "${REGISTRY_IMAGE}@${AMD64_DIGEST}"
-            "${REGISTRY_IMAGE}@${ARM64_DIGEST}"
-          )
-
-          if [[ "$TAG" =~ ^cli/v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              -t "${REGISTRY_IMAGE}:latest" \
-              "${IMAGES[@]}"
-          else
-            docker buildx imagetools create \
-              -t "${REGISTRY_IMAGE}:${SANITIZED_TAG}" \
-              "${IMAGES[@]}"
-          fi

.github/workflows/release-devtools.yml

@@ -22,11 +22,13 @@ jobs:
           - { goos: "windows", goarch: "arm64" }
           - { goos: "darwin", goarch: "amd64" }
           - { goos: "darwin", goarch: "arm64" }
+          - { goos: "", goarch: "" }
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
         with:
           persist-credentials: false
-      - uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+          fetch-depth: 0
+      - uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"

.github/workflows/reusable-nightly-llm-provider-chat.yml

@@ -1,333 +0,0 @@
-name: Reusable Nightly LLM Provider Chat Tests
-
-on:
-  workflow_call:
-    inputs:
-      openai_models:
-        description: "Comma-separated models for openai"
-        required: false
-        default: ""
-        type: string
-      anthropic_models:
-        description: "Comma-separated models for anthropic"
-        required: false
-        default: ""
-        type: string
-      bedrock_models:
-        description: "Comma-separated models for bedrock"
-        required: false
-        default: ""
-        type: string
-      vertex_ai_models:
-        description: "Comma-separated models for vertex_ai"
-        required: false
-        default: ""
-        type: string
-      azure_models:
-        description: "Comma-separated models for azure"
-        required: false
-        default: ""
-        type: string
-      ollama_models:
-        description: "Comma-separated models for ollama_chat"
-        required: false
-        default: ""
-        type: string
-      openrouter_models:
-        description: "Comma-separated models for openrouter"
-        required: false
-        default: ""
-        type: string
-      azure_api_base:
-        description: "API base for azure provider"
-        required: false
-        default: ""
-        type: string
-      strict:
-        description: "Default NIGHTLY_LLM_STRICT passed to tests"
-        required: false
-        default: true
-        type: boolean
-    secrets:
-      AWS_OIDC_ROLE_ARN:
-        description: "AWS role ARN for OIDC auth"
-        required: true
-
-permissions:
-  contents: read
-  id-token: write
-
-jobs:
-  build-backend-image:
-    runs-on:
-      [
-        runs-on,
-        runner=1cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-build-backend-image",
-        "extras=ecr-cache",
-      ]
-    timeout-minutes: 45
-    environment: ci-protected
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
-      - name: Build backend image
-        uses: ./.github/actions/build-backend-image
-        with:
-          runs-on-ecr-cache: ${{ env.RUNS_ON_ECR_CACHE }}
-          ref-name: ${{ github.ref_name }}
-          pr-number: ${{ github.event.pull_request.number }}
-          github-sha: ${{ github.sha }}
-          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
-          docker-no-cache: ${{ vars.DOCKER_NO_CACHE == 'true' && 'true' || 'false' }}
-
-  build-model-server-image:
-    runs-on:
-      [
-        runs-on,
-        runner=1cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-build-model-server-image",
-        "extras=ecr-cache",
-      ]
-    timeout-minutes: 45
-    environment: ci-protected
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
-      - name: Build model server image
-        uses: ./.github/actions/build-model-server-image
-        with:
-          runs-on-ecr-cache: ${{ env.RUNS_ON_ECR_CACHE }}
-          ref-name: ${{ github.ref_name }}
-          pr-number: ${{ github.event.pull_request.number }}
-          github-sha: ${{ github.sha }}
-          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
-
-  build-integration-image:
-    runs-on:
-      [
-        runs-on,
-        runner=2cpu-linux-arm64,
-        "run-id=${{ github.run_id }}-build-integration-image",
-        "extras=ecr-cache",
-      ]
-    timeout-minutes: 45
-    environment: ci-protected
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-
-      - name: Build integration image
-        uses: ./.github/actions/build-integration-image
-        with:
-          runs-on-ecr-cache: ${{ env.RUNS_ON_ECR_CACHE }}
-          ref-name: ${{ github.ref_name }}
-          pr-number: ${{ github.event.pull_request.number }}
-          github-sha: ${{ github.sha }}
-          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
-
-  provider-chat-test:
-    needs:
-      [
-        build-backend-image,
-        build-model-server-image,
-        build-integration-image,
-      ]
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - provider: openai
-            models: ${{ inputs.openai_models }}
-            api_key_env: OPENAI_API_KEY
-            custom_config_env: ""
-            api_base: ""
-            api_version: ""
-            deployment_name: ""
-            required: true
-          - provider: anthropic
-            models: ${{ inputs.anthropic_models }}
-            api_key_env: ANTHROPIC_API_KEY
-            custom_config_env: ""
-            api_base: ""
-            api_version: ""
-            deployment_name: ""
-            required: true
-          - provider: bedrock
-            models: ${{ inputs.bedrock_models }}
-            api_key_env: BEDROCK_API_KEY
-            custom_config_env: ""
-            api_base: ""
-            api_version: ""
-            deployment_name: ""
-            required: false
-          - provider: vertex_ai
-            models: ${{ inputs.vertex_ai_models }}
-            api_key_env: ""
-            custom_config_env: NIGHTLY_LLM_VERTEX_AI_CUSTOM_CONFIG_JSON
-            api_base: ""
-            api_version: ""
-            deployment_name: ""
-            required: false
-          - provider: azure
-            models: ${{ inputs.azure_models }}
-            api_key_env: AZURE_API_KEY
-            custom_config_env: ""
-            api_base: ${{ inputs.azure_api_base }}
-            api_version: "2025-04-01-preview"
-            deployment_name: ""
-            required: false
-          - provider: ollama_chat
-            models: ${{ inputs.ollama_models }}
-            api_key_env: OLLAMA_API_KEY
-            custom_config_env: ""
-            api_base: "https://ollama.com"
-            api_version: ""
-            deployment_name: ""
-            required: false
-          - provider: openrouter
-            models: ${{ inputs.openrouter_models }}
-            api_key_env: OPENROUTER_API_KEY
-            custom_config_env: ""
-            api_base: "https://openrouter.ai/api/v1"
-            api_version: ""
-            deployment_name: ""
-            required: false
-    runs-on:
-      - runs-on
-      - runner=4cpu-linux-arm64
-      - "run-id=${{ github.run_id }}-nightly-${{ matrix.provider }}-provider-chat-test"
-      - extras=ecr-cache
-    timeout-minutes: 45
-    environment: ci-protected
-    steps:
-      - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
-
-      - name: Checkout code
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v6
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
-        with:
-          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
-          aws-region: us-east-2
-
-      - name: Get AWS Secrets
-        uses: aws-actions/aws-secretsmanager-get-secrets@a9a7eb4e2f2871d30dc5b892576fde60a2ecc802
-        with:
-          # Keep JSON values unparsed so vertex custom config is passed as raw JSON.
-          parse-json-secrets: false
-          secret-ids: |
-            DOCKER_USERNAME, test/docker-username
-            DOCKER_TOKEN, test/docker-token
-            OPENAI_API_KEY, test/openai-api-key
-            ANTHROPIC_API_KEY, test/anthropic-api-key
-            BEDROCK_API_KEY, test/bedrock-api-key
-            NIGHTLY_LLM_VERTEX_AI_CUSTOM_CONFIG_JSON, test/nightly-llm-vertex-ai-custom-config-json
-            AZURE_API_KEY, test/azure-api-key
-            OLLAMA_API_KEY, test/ollama-api-key
-            OPENROUTER_API_KEY, test/openrouter-api-key
-
-      - name: Run nightly provider chat test
-        uses: ./.github/actions/run-nightly-provider-chat-test
-        with:
-          provider: ${{ matrix.provider }}
-          models: ${{ matrix.models }}
-          provider-api-key: ${{ matrix.api_key_env && env[matrix.api_key_env] || '' }}
-          strict: ${{ inputs.strict && 'true' || 'false' }}
-          api-base: ${{ matrix.api_base }}
-          api-version: ${{ matrix.api_version }}
-          deployment-name: ${{ matrix.deployment_name }}
-          custom-config-json: ${{ matrix.custom_config_env && env[matrix.custom_config_env] || '' }}
-          runs-on-ecr-cache: ${{ env.RUNS_ON_ECR_CACHE }}
-          run-id: ${{ github.run_id }}
-          docker-username: ${{ env.DOCKER_USERNAME }}
-          docker-token: ${{ env.DOCKER_TOKEN }}
-
-      - name: Dump API server logs
-        if: always()
-        run: |
-          cd deployment/docker_compose
-          docker compose logs --no-color api_server > $GITHUB_WORKSPACE/api_server.log || true
-
-      - name: Dump all-container logs
-        if: always()
-        run: |
-          cd deployment/docker_compose
-          docker compose logs --no-color > $GITHUB_WORKSPACE/docker-compose.log || true
-
-      - name: Upload logs
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
-        with:
-          name: docker-all-logs-nightly-${{ matrix.provider }}-llm-provider
-          path: |
-            ${{ github.workspace }}/api_server.log
-            ${{ github.workspace }}/docker-compose.log
-
-      - name: Stop Docker containers
-        if: always()
-        run: |
-          cd deployment/docker_compose
-          docker compose down -v

.github/workflows/sandbox-deployment.yml

@@ -110,7 +110,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -180,7 +180,7 @@ jobs:
           persist-credentials: false
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2
@@ -244,7 +244,7 @@ jobs:
       - uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # ratchet:runs-on/action@v2
 
       - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7
+        uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708
         with:
           role-to-assume: ${{ secrets.AWS_OIDC_ROLE_ARN }}
           aws-region: us-east-2

.github/workflows/storybook-deploy.yml

@@ -1,69 +0,0 @@
-name: Storybook Deploy
-env:
-  VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
-  VERCEL_PROJECT_ID: prj_sG49mVsA25UsxIPhN2pmBJlikJZM
-  VERCEL_CLI: vercel@50.14.1
-  VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
-
-concurrency:
-  group: storybook-deploy-production
-  cancel-in-progress: true
-
-on:
-  workflow_dispatch:
-  push:
-    branches:
-      - main
-    paths:
-      - "web/lib/opal/**"
-      - "web/src/refresh-components/**"
-      - "web/.storybook/**"
-      - "web/package.json"
-      - "web/package-lock.json"
-permissions:
-  contents: read
-jobs:
-  Deploy-Storybook:
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v4
-        with:
-          persist-credentials: false
-
-      - name: Setup node
-        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # ratchet:actions/setup-node@v4
-        with:
-          node-version: 22
-          cache: "npm"
-          cache-dependency-path: ./web/package-lock.json
-
-      - name: Install dependencies
-        working-directory: web
-        run: npm ci
-
-      - name: Build Storybook
-        working-directory: web
-        run: npm run storybook:build
-
-      - name: Deploy to Vercel (Production)
-        working-directory: web
-        run: npx --yes "$VERCEL_CLI" deploy storybook-static/ --prod --yes --token="$VERCEL_TOKEN"
-
-  notify-slack-on-failure:
-    needs: Deploy-Storybook
-    if: always() && needs.Deploy-Storybook.result == 'failure'
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # ratchet:actions/checkout@v4
-        with:
-          persist-credentials: false
-          sparse-checkout: .github/actions/slack-notify
-
-      - name: Send Slack notification
-        uses: ./.github/actions/slack-notify
-        with:
-          webhook-url: ${{ secrets.MONITOR_DEPLOYMENTS_WEBHOOK }}
-          failed-jobs: "• Deploy-Storybook"
-          title: "🚨 Storybook Deploy Failed"

.github/workflows/zizmor.yml

@@ -5,8 +5,6 @@ on:
     branches: ["main"]
   pull_request:
     branches: ["**"]
-    paths:
-      - ".github/**"
 
 permissions: {}
 
@@ -23,18 +21,29 @@ jobs:
         with:
           persist-credentials: false
 
+      - name: Detect changes
+        id: filter
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # ratchet:dorny/paths-filter@v3
+        with:
+          filters: |
+            zizmor:
+              - '.github/**'
+
       - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@5a095e7a2014a4212f075830d4f7277575a9d098 # ratchet:astral-sh/setup-uv@v7
+        if: steps.filter.outputs.zizmor == 'true' || github.ref_name == 'main'
+        uses: astral-sh/setup-uv@61cb8a9741eeb8a550a1b8544337180c0fc8476b # ratchet:astral-sh/setup-uv@v7
         with:
           enable-cache: false
           version: "0.9.9"
 
       - name: Run zizmor
+        if: steps.filter.outputs.zizmor == 'true' || github.ref_name == 'main'
         run: uv run --no-sync --with zizmor zizmor --format=sarif . > results.sarif
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Upload SARIF file
+        if: steps.filter.outputs.zizmor == 'true' || github.ref_name == 'main'
         uses: github/codeql-action/upload-sarif@ba454b8ab46733eb6145342877cd148270bb77ab # ratchet:github/codeql-action/upload-sarif@codeql-bundle-v2.23.5
         with:
           sarif_file: results.sarif

.gitignore

@@ -7,7 +7,6 @@
 .zed
 .cursor
 !/.cursor/mcp.json
-!/.cursor/skills/
 
 # macos
 .DS_store

.pre-commit-config.yaml

@@ -119,11 +119,10 @@ repos:
           ]
 
   - repo: https://github.com/golangci/golangci-lint
-    rev: 5d1e709b7be35cb2025444e19de266b056b7b7ee # frozen: v2.10.1
+    rev: 9f61b0f53f80672872fced07b6874397c3ed197b # frozen: v2.7.2
     hooks:
       - id: golangci-lint
-        language_version: "1.26.0"
-        entry: bash -c "find . -name go.mod -not -path './.venv/*' -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"
+        entry: bash -c "find tools/ -name go.mod -print0 | xargs -0 -I{} bash -c 'cd \"$(dirname {})\" && golangci-lint run ./...'"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.

.vscode/env_template.txt

@@ -7,9 +7,6 @@
 
 
 AUTH_TYPE=basic
-# Recommended for basic auth - used for signing password reset and verification tokens
-# Generate a secure value with: openssl rand -hex 32
-USER_AUTH_SECRET=""
 DEV_MODE=true
 
 

.vscode/launch.json

@@ -40,7 +40,19 @@
       }
     },
     {
-      "name": "Celery",
+      "name": "Celery (lightweight mode)",
+      "configurations": [
+        "Celery primary",
+        "Celery background",
+        "Celery beat"
+      ],
+      "presentation": {
+        "group": "1"
+      },
+      "stopAll": true
+    },
+    {
+      "name": "Celery (standard mode)",
       "configurations": [
         "Celery primary",
         "Celery light",
@@ -241,6 +253,35 @@
       },
       "consoleTitle": "Celery light Console"
     },
+    {
+      "name": "Celery background",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "celery",
+      "cwd": "${workspaceFolder}/backend",
+      "envFile": "${workspaceFolder}/.vscode/.env",
+      "env": {
+        "LOG_LEVEL": "INFO",
+        "PYTHONUNBUFFERED": "1",
+        "PYTHONPATH": "."
+      },
+      "args": [
+        "-A",
+        "onyx.background.celery.versioned_apps.background",
+        "worker",
+        "--pool=threads",
+        "--concurrency=20",
+        "--prefetch-multiplier=4",
+        "--loglevel=INFO",
+        "--hostname=background@%n",
+        "-Q",
+        "vespa_metadata_sync,connector_deletion,doc_permissions_upsert,checkpoint_cleanup,index_attempt_cleanup,docprocessing,connector_doc_fetching,connector_pruning,connector_doc_permissions_sync,connector_external_group_sync,csv_generation,kg_processing,monitoring,user_file_processing,user_file_project_sync,user_file_delete,opensearch_migration"
+      ],
+      "presentation": {
+        "group": "2"
+      },
+      "consoleTitle": "Celery background Console"
+    },
     {
       "name": "Celery heavy",
       "type": "debugpy",
@@ -485,6 +526,21 @@
         "group": "3"
       }
     },
+    {
+      "name": "Clear and Restart OpenSearch Container",
+      // Generic debugger type, required arg but has no bearing on bash.
+      "type": "node",
+      "request": "launch",
+      "runtimeExecutable": "bash",
+      "runtimeArgs": [
+        "${workspaceFolder}/backend/scripts/restart_opensearch_container.sh"
+      ],
+      "cwd": "${workspaceFolder}",
+      "console": "integratedTerminal",
+      "presentation": {
+        "group": "3"
+      }
+    },
     {
       "name": "Eval CLI",
       "type": "debugpy",

AGENTS.md

@@ -86,6 +86,37 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work
      - Monitoring tasks (every 5 minutes)
      - Cleanup tasks (hourly)
 
+#### Worker Deployment Modes
+
+Onyx supports two deployment modes for background workers, controlled by the `USE_LIGHTWEIGHT_BACKGROUND_WORKER` environment variable:
+
+**Lightweight Mode** (default, `USE_LIGHTWEIGHT_BACKGROUND_WORKER=true`):
+
+- Runs a single consolidated `background` worker that handles all background tasks:
+  - Light worker tasks (Vespa operations, permissions sync, deletion)
+  - Document processing (indexing pipeline)
+  - Document fetching (connector data retrieval)
+  - Pruning operations (from `heavy` worker)
+  - Knowledge graph processing (from `kg_processing` worker)
+  - Monitoring tasks (from `monitoring` worker)
+  - User file processing (from `user_file_processing` worker)
+- Lower resource footprint (fewer worker processes)
+- Suitable for smaller deployments or development environments
+- Default concurrency: 20 threads (increased to handle combined workload)
+
+**Standard Mode** (`USE_LIGHTWEIGHT_BACKGROUND_WORKER=false`):
+
+- Runs separate specialized workers as documented above (light, docprocessing, docfetching, heavy, kg_processing, monitoring, user_file_processing)
+- Better isolation and scalability
+- Can scale individual workers independently based on workload
+- Suitable for production deployments with higher load
+
+The deployment mode affects:
+
+- **Backend**: Worker processes spawned by supervisord or dev scripts
+- **Helm**: Which Kubernetes deployments are created
+- **Dev Environment**: Which workers `dev_run_background_jobs.py` spawns
+
 #### Key Features
 
 - **Thread-based Workers**: All workers use thread pools (not processes) for stability
@@ -104,10 +135,6 @@ Onyx uses Celery for asynchronous task processing with multiple specialized work
 
 - Always use `@shared_task` rather than `@celery_app`
 - Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks`
-- Never enqueue a task without an expiration. Always supply `expires=` when
-  sending tasks, either from the beat schedule or directly from another task. It
-  should never be acceptable to submit code which enqueues tasks without an
-  expiration, as doing so can lead to unbounded task queue growth.
 
 **Defining APIs**:
 When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the
@@ -521,7 +548,7 @@ class in the utils over directly calling the APIs with a library like `requests`
 calling the utilities directly (e.g. do NOT create admin users with
 `admin_user = UserManager.create(name="admin_user")`, instead use the `admin_user` fixture).
 
-A great example of this type of test is `backend/tests/integration/tests/streaming_endpoints/test_chat_stream.py`.
+A great example of this type of test is `backend/tests/integration/dev_apis/test_simple_chat_api.py`.
 
 To run them:
 
@@ -544,8 +571,6 @@ To run them:
 npx playwright test <TEST_NAME>
 ```
 
-For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`.
-
 ## Logs
 
 When (1) writing integration tests or (2) doing live tests (e.g. curl / playwright) you can get access
@@ -591,48 +616,3 @@ This is a minimal list - feel free to include more. Do NOT write code as part of
 Keep it high level. You can reference certain files or functions though.
 
 Before writing your plan, make sure to do research. Explore the relevant sections in the codebase.
-
-## Error Handling
-
-**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`.
-Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.**
-
-A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard
-`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error
-handling consistent across the entire backend.
-
-```python
-from onyx.error_handling.error_codes import OnyxErrorCode
-from onyx.error_handling.exceptions import OnyxError
-
-# ✅ Good
-raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found")
-
-# ✅ Good — no extra message needed
-raise OnyxError(OnyxErrorCode.UNAUTHENTICATED)
-
-# ✅ Good — upstream service with dynamic status code
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status)
-
-# ❌ Bad — using HTTPException directly
-raise HTTPException(status_code=404, detail="Session not found")
-
-# ❌ Bad — starlette constant
-raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
-```
-
-Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error
-category is needed, add it there first — do not invent ad-hoc codes.
-
-**Upstream service errors:** When forwarding errors from an upstream service where the HTTP
-status code is dynamic (comes from the upstream response), use `status_code_override`:
-
-```python
-raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code)
-```
-
-## Best Practices
-
-In addition to the other content in this file, best practices for contributing
-to the codebase can be found at `contributing_guides/best_practices.md`.
-Understand its contents and follow them.

backend/Dockerfile

@@ -46,9 +46,7 @@ RUN apt-get update && \
         pkg-config \
         gcc \
         nano \
-        vim \
-        libjemalloc2 \
-        && \
+        vim && \
     rm -rf /var/lib/apt/lists/* && \
     apt-get clean
 
@@ -143,7 +141,6 @@ COPY --chown=onyx:onyx ./scripts/debugging /app/scripts/debugging
 COPY --chown=onyx:onyx ./scripts/force_delete_connector_by_id.py /app/scripts/force_delete_connector_by_id.py
 COPY --chown=onyx:onyx ./scripts/supervisord_entrypoint.sh /app/scripts/supervisord_entrypoint.sh
 COPY --chown=onyx:onyx ./scripts/setup_craft_templates.sh /app/scripts/setup_craft_templates.sh
-COPY --chown=onyx:onyx ./scripts/reencrypt_secrets.py /app/scripts/reencrypt_secrets.py
 RUN chmod +x /app/scripts/supervisord_entrypoint.sh /app/scripts/setup_craft_templates.sh
 
 # Run Craft template setup at build time when ENABLE_CRAFT=true
@@ -167,13 +164,6 @@ ENV PYTHONPATH=/app
 ARG ONYX_VERSION=0.0.0-dev
 ENV ONYX_VERSION=${ONYX_VERSION}
 
-# Use jemalloc instead of glibc malloc to reduce memory fragmentation
-# in long-running Python processes (API server, Celery workers).
-# The soname is architecture-independent; the dynamic linker resolves
-# the correct path from standard library directories.
-# Placed after all RUN steps so build-time processes are unaffected.
-ENV LD_PRELOAD=libjemalloc.so.2
-
 # Default command which does nothing
 # This container is used by api server and background which specify their own CMD
 CMD ["tail", "-f", "/dev/null"]

backend/alembic/env.py

@@ -244,10 +244,7 @@ def do_run_migrations(
 
 
 def provide_iam_token_for_alembic(
-    dialect: Any,  # noqa: ARG001
-    conn_rec: Any,  # noqa: ARG001
-    cargs: Any,  # noqa: ARG001
-    cparams: Any,
+    dialect: Any, conn_rec: Any, cargs: Any, cparams: Any  # noqa: ARG001
 ) -> None:
     if USE_IAM_AUTH:
         # Database connection settings
@@ -363,7 +360,8 @@ async def run_async_migrations() -> None:
         # upgrade_all_tenants=true or schemas in multi-tenant mode
         # and for non-multi-tenant mode, we should use schemas with the default schema
         raise ValueError(
-            "No migration target specified. Use either upgrade_all_tenants=true for all tenants or schemas for specific schemas."
+            "No migration target specified. Use either upgrade_all_tenants=true for all tenants "
+            "or schemas for specific schemas."
         )
 
     await engine.dispose()
@@ -459,7 +457,8 @@ def run_migrations_offline() -> None:
     else:
         # This should not happen in the new design
         raise ValueError(
-            "No migration target specified. Use either upgrade_all_tenants=true for all tenants or schemas for specific schemas."
+            "No migration target specified. Use either upgrade_all_tenants=true for all tenants "
+            "or schemas for specific schemas."
         )
 
 

backend/alembic/run_multitenant_migrations.py

@@ -13,7 +13,6 @@ Usage examples::
     # custom settings
     python alembic/run_multitenant_migrations.py -j 8 -b 100
 """
-
 from __future__ import annotations
 
 import argparse
@@ -22,14 +21,15 @@ import sys
 import threading
 import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import NamedTuple
+from typing import List, NamedTuple
 
 from alembic.config import Config
 from alembic.script import ScriptDirectory
+from sqlalchemy import text
 
+from onyx.db.engine.sql_engine import is_valid_schema_name
 from onyx.db.engine.sql_engine import SqlEngine
 from onyx.db.engine.tenant_utils import get_all_tenant_ids
-from onyx.db.engine.tenant_utils import get_schemas_needing_migration
 from shared_configs.configs import TENANT_ID_PREFIX
 
 
@@ -105,6 +105,56 @@ def get_head_revision() -> str | None:
     return script.get_current_head()
 
 
+def get_schemas_needing_migration(
+    tenant_schemas: List[str], head_rev: str
+) -> List[str]:
+    """Return only schemas whose current alembic version is not at head."""
+    if not tenant_schemas:
+        return []
+
+    engine = SqlEngine.get_engine()
+
+    with engine.connect() as conn:
+        # Find which schemas actually have an alembic_version table
+        rows = conn.execute(
+            text(
+                "SELECT table_schema FROM information_schema.tables "
+                "WHERE table_name = 'alembic_version' "
+                "AND table_schema = ANY(:schemas)"
+            ),
+            {"schemas": tenant_schemas},
+        )
+        schemas_with_table = set(row[0] for row in rows)
+
+        # Schemas without the table definitely need migration
+        needs_migration = [s for s in tenant_schemas if s not in schemas_with_table]
+
+        if not schemas_with_table:
+            return needs_migration
+
+        # Validate schema names before interpolating into SQL
+        for schema in schemas_with_table:
+            if not is_valid_schema_name(schema):
+                raise ValueError(f"Invalid schema name: {schema}")
+
+        # Single query to get every schema's current revision at once.
+        # Use integer tags instead of interpolating schema names into
+        # string literals to avoid quoting issues.
+        schema_list = list(schemas_with_table)
+        union_parts = [
+            f'SELECT {i} AS idx, version_num FROM "{schema}".alembic_version'
+            for i, schema in enumerate(schema_list)
+        ]
+        rows = conn.execute(text(" UNION ALL ".join(union_parts)))
+        version_by_schema = {schema_list[row[0]]: row[1] for row in rows}
+
+        needs_migration.extend(
+            s for s in schemas_with_table if version_by_schema.get(s) != head_rev
+        )
+
+    return needs_migration
+
+
 def run_migrations_parallel(
     schemas: list[str],
     max_workers: int,
@@ -118,7 +168,8 @@ def run_migrations_parallel(
     batches = [schemas[i : i + batch_size] for i in range(0, len(schemas), batch_size)]
     total_batches = len(batches)
     print(
-        f"{len(schemas)} schemas in {total_batches} batch(es) with {max_workers} workers (batch size: {batch_size})...",
+        f"{len(schemas)} schemas in {total_batches} batch(es) "
+        f"with {max_workers} workers (batch size: {batch_size})...",
         flush=True,
     )
     all_success = True
@@ -166,7 +217,8 @@ def run_migrations_parallel(
                 with lock:
                     in_flight[batch_idx] = batch
                 print(
-                    f"Batch {batch_idx + 1}/{total_batches} started ({len(batch)} schemas): {', '.join(batch)}",
+                    f"Batch {batch_idx + 1}/{total_batches} started "
+                    f"({len(batch)} schemas): {', '.join(batch)}",
                     flush=True,
                 )
                 result = run_alembic_for_batch(batch)
@@ -200,7 +252,7 @@ def run_migrations_parallel(
 
                 except Exception as e:
                     print(
-                        f"Batch {batch_idx + 1}/{total_batches} ✗ exception: {e}",
+                        f"Batch {batch_idx + 1}/{total_batches} " f"✗ exception: {e}",
                         flush=True,
                     )
                     all_success = False
@@ -267,12 +319,14 @@ def main() -> int:
 
     if not schemas_to_migrate:
         print(
-            f"All {len(tenant_schemas)} tenants are already at head revision ({head_rev})."
+            f"All {len(tenant_schemas)} tenants are already at head "
+            f"revision ({head_rev})."
         )
         return 0
 
     print(
-        f"{len(schemas_to_migrate)}/{len(tenant_schemas)} tenants need migration (head: {head_rev})."
+        f"{len(schemas_to_migrate)}/{len(tenant_schemas)} tenants need "
+        f"migration (head: {head_rev})."
     )
 
     success = run_migrations_parallel(

backend/alembic/versions/07b98176f1de_code_interpreter_seed.py

@@ -1,29 +0,0 @@
-"""code interpreter seed
-
-Revision ID: 07b98176f1de
-Revises: 7cb492013621
-Create Date: 2026-02-23 15:55:07.606784
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "07b98176f1de"
-down_revision = "7cb492013621"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Seed the single instance of code_interpreter_server
-    # NOTE: There should only exist at most and at minimum 1 code_interpreter_server row
-    op.execute(
-        sa.text("INSERT INTO code_interpreter_server (server_enabled) VALUES (true)")
-    )
-
-
-def downgrade() -> None:
-    op.execute(sa.text("DELETE FROM code_interpreter_server"))

backend/alembic/versions/0bb4558f35df_add_scim_username_to_scim_user_mapping.py

@@ -1,28 +0,0 @@
-"""add scim_username to scim_user_mapping
-
-Revision ID: 0bb4558f35df
-Revises: 631fd2504136
-Create Date: 2026-02-20 10:45:30.340188
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "0bb4558f35df"
-down_revision = "631fd2504136"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("scim_username", sa.String(), nullable=True),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("scim_user_mapping", "scim_username")

backend/alembic/versions/2664261bfaab_add_cache_store_table.py

@@ -1,37 +0,0 @@
-"""add cache_store table
-
-Revision ID: 2664261bfaab
-Revises: 4a1e4b1c89d2
-Create Date: 2026-02-27 00:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-# revision identifiers, used by Alembic.
-revision = "2664261bfaab"
-down_revision = "4a1e4b1c89d2"
-branch_labels: None = None
-depends_on: None = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "cache_store",
-        sa.Column("key", sa.String(), nullable=False),
-        sa.Column("value", sa.LargeBinary(), nullable=True),
-        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
-        sa.PrimaryKeyConstraint("key"),
-    )
-    op.create_index(
-        "ix_cache_store_expires",
-        "cache_store",
-        ["expires_at"],
-        postgresql_where=sa.text("expires_at IS NOT NULL"),
-    )
-
-
-def downgrade() -> None:
-    op.drop_index("ix_cache_store_expires", table_name="cache_store")
-    op.drop_table("cache_store")

backend/alembic/versions/27fb147a843f_add_timestamps_to_user_table.py

@@ -1,43 +0,0 @@
-"""add timestamps to user table
-
-Revision ID: 27fb147a843f
-Revises: b5c4d7e8f9a1
-Create Date: 2026-03-08 17:18:40.828644
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "27fb147a843f"
-down_revision = "b5c4d7e8f9a1"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "user",
-        sa.Column(
-            "created_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "updated_at",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("user", "updated_at")
-    op.drop_column("user", "created_at")

backend/alembic/versions/2b75d0a8ffcb_user_file_schema_cleanup.py

@@ -50,7 +50,8 @@ def upgrade() -> None:
 
         if orphaned_count > 0:
             logger.warning(
-                f"WARNING: {orphaned_count} chat_session records still have folder_id without project_id. Proceeding anyway."
+                f"WARNING: {orphaned_count} chat_session records still have "
+                f"folder_id without project_id. Proceeding anyway."
             )
 
     # === Step 2: Drop chat_session.folder_id ===

backend/alembic/versions/3a78dba1080a_user_file_legacy_data_cleanup.py

@@ -75,7 +75,8 @@ def batch_delete(
 
     if failed_batches:
         logger.warning(
-            f"Failed to delete {len(failed_batches)} batches from {table_name}. Total deleted: {total_deleted}/{total_count}"
+            f"Failed to delete {len(failed_batches)} batches from {table_name}. "
+            f"Total deleted: {total_deleted}/{total_count}"
         )
         # Fail the migration to avoid silently succeeding on partial cleanup
         raise RuntimeError(

backend/alembic/versions/40926a4dab77_reset_userfile_document_id_migrated_.py

@@ -18,7 +18,8 @@ depends_on = None
 def upgrade() -> None:
     # Set all existing records to not migrated
     op.execute(
-        "UPDATE user_file SET document_id_migrated = FALSE WHERE document_id_migrated IS DISTINCT FROM FALSE;"
+        "UPDATE user_file SET document_id_migrated = FALSE "
+        "WHERE document_id_migrated IS DISTINCT FROM FALSE;"
     )
 
 

backend/alembic/versions/495cb26ce93e_create_knowlege_graph_tables.py

@@ -35,6 +35,7 @@ def upgrade() -> None:
     # environment variables MUST be set. Otherwise, an exception will be raised.
 
     if not MULTI_TENANT:
+
         # Enable pg_trgm extension if not already enabled
         op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
 
@@ -480,7 +481,8 @@ def upgrade() -> None:
         f"ON kg_entity USING GIN (name {POSTGRES_DEFAULT_SCHEMA}.gin_trgm_ops)"
     )
     op.execute(
-        "CREATE INDEX IF NOT EXISTS idx_kg_entity_normalization_trigrams ON kg_entity USING GIN (name_trigrams)"
+        "CREATE INDEX IF NOT EXISTS idx_kg_entity_normalization_trigrams "
+        "ON kg_entity USING GIN (name_trigrams)"
     )
 
     # Create kg_entity trigger to update kg_entity.name and its trigrams

backend/alembic/versions/4a1e4b1c89d2_add_indexing_to_userfilestatus.py

@@ -1,51 +0,0 @@
-"""Add INDEXING to UserFileStatus
-
-Revision ID: 4a1e4b1c89d2
-Revises: 6b3b4083c5aa
-Create Date: 2026-02-28 00:00:00.000000
-
-"""
-
-import sqlalchemy as sa
-from alembic import op
-
-revision = "4a1e4b1c89d2"
-down_revision = "6b3b4083c5aa"
-branch_labels = None
-depends_on = None
-
-TABLE = "user_file"
-COLUMN = "status"
-CONSTRAINT_NAME = "ck_user_file_status"
-
-OLD_VALUES = ("PROCESSING", "COMPLETED", "FAILED", "CANCELED", "DELETING")
-NEW_VALUES = ("PROCESSING", "INDEXING", "COMPLETED", "FAILED", "CANCELED", "DELETING")
-
-
-def _drop_status_check_constraint() -> None:
-    """Drop the existing CHECK constraint on user_file.status.
-
-    The constraint name is auto-generated by SQLAlchemy and unknown,
-    so we look it up via the inspector.
-    """
-    inspector = sa.inspect(op.get_bind())
-    for constraint in inspector.get_check_constraints(TABLE):
-        if COLUMN in constraint.get("sqltext", ""):
-            constraint_name = constraint["name"]
-            if constraint_name is not None:
-                op.drop_constraint(constraint_name, TABLE, type_="check")
-
-
-def upgrade() -> None:
-    _drop_status_check_constraint()
-    in_clause = ", ".join(f"'{v}'" for v in NEW_VALUES)
-    op.create_check_constraint(CONSTRAINT_NAME, TABLE, f"{COLUMN} IN ({in_clause})")
-
-
-def downgrade() -> None:
-    op.execute(
-        f"UPDATE {TABLE} SET {COLUMN} = 'PROCESSING' WHERE {COLUMN} = 'INDEXING'"
-    )
-    op.drop_constraint(CONSTRAINT_NAME, TABLE, type_="check")
-    in_clause = ", ".join(f"'{v}'" for v in OLD_VALUES)
-    op.create_check_constraint(CONSTRAINT_NAME, TABLE, f"{COLUMN} IN ({in_clause})")

backend/alembic/versions/4d58345da04a_lowercase_user_emails.py

@@ -51,7 +51,10 @@ def upgrade() -> None:
                 next_email = f"{username.lower()}_{attempt}@{domain.lower()}"
                 # Email conflict occurred, append `_1`, `_2`, etc., to the username
                 logger.warning(
-                    f"Conflict while lowercasing email: old_email={email} conflicting_email={new_email} next_email={next_email}"
+                    f"Conflict while lowercasing email: "
+                    f"old_email={email} "
+                    f"conflicting_email={new_email} "
+                    f"next_email={next_email}"
                 )
                 new_email = next_email
                 attempt += 1

backend/alembic/versions/57122d037335_add_python_tool_on_default.py

@@ -1,69 +0,0 @@
-"""add python tool on default
-
-Revision ID: 57122d037335
-Revises: c0c937d5c9e5
-Create Date: 2026-02-27 10:10:40.124925
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "57122d037335"
-down_revision = "c0c937d5c9e5"
-branch_labels = None
-depends_on = None
-
-
-PYTHON_TOOL_NAME = "python"
-
-
-def upgrade() -> None:
-    conn = op.get_bind()
-
-    # Look up the PythonTool id
-    result = conn.execute(
-        sa.text("SELECT id FROM tool WHERE name = :name"),
-        {"name": PYTHON_TOOL_NAME},
-    ).fetchone()
-
-    if not result:
-        return
-
-    tool_id = result[0]
-
-    # Attach to the default persona (id=0) if not already attached
-    conn.execute(
-        sa.text(
-            """
-            INSERT INTO persona__tool (persona_id, tool_id)
-            VALUES (0, :tool_id)
-            ON CONFLICT DO NOTHING
-            """
-        ),
-        {"tool_id": tool_id},
-    )
-
-
-def downgrade() -> None:
-    conn = op.get_bind()
-
-    result = conn.execute(
-        sa.text("SELECT id FROM tool WHERE name = :name"),
-        {"name": PYTHON_TOOL_NAME},
-    ).fetchone()
-
-    if not result:
-        return
-
-    conn.execute(
-        sa.text(
-            """
-            DELETE FROM persona__tool
-            WHERE persona_id = 0 AND tool_id = :tool_id
-            """
-        ),
-        {"tool_id": result[0]},
-    )

backend/alembic/versions/631fd2504136_add_approx_chunk_count_in_vespa_to_.py

@@ -1,32 +0,0 @@
-"""add approx_chunk_count_in_vespa to opensearch tenant migration
-
-Revision ID: 631fd2504136
-Revises: c7f2e1b4a9d3
-Create Date: 2026-02-18 21:07:52.831215
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "631fd2504136"
-down_revision = "c7f2e1b4a9d3"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "opensearch_tenant_migration_record",
-        sa.Column(
-            "approx_chunk_count_in_vespa",
-            sa.Integer(),
-            nullable=True,
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("opensearch_tenant_migration_record", "approx_chunk_count_in_vespa")

backend/alembic/versions/6b3b4083c5aa_persona_cleanup_and_featured.py

@@ -1,112 +0,0 @@
-"""persona cleanup and featured
-
-Revision ID: 6b3b4083c5aa
-Revises: 57122d037335
-Create Date: 2026-02-26 12:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "6b3b4083c5aa"
-down_revision = "57122d037335"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Add featured column with nullable=True first
-    op.add_column("persona", sa.Column("featured", sa.Boolean(), nullable=True))
-
-    # Migrate data from is_default_persona to featured
-    op.execute("UPDATE persona SET featured = is_default_persona")
-
-    # Make featured non-nullable with default=False
-    op.alter_column(
-        "persona",
-        "featured",
-        existing_type=sa.Boolean(),
-        nullable=False,
-        server_default=sa.false(),
-    )
-
-    # Drop is_default_persona column
-    op.drop_column("persona", "is_default_persona")
-
-    # Drop unused columns
-    op.drop_column("persona", "num_chunks")
-    op.drop_column("persona", "chunks_above")
-    op.drop_column("persona", "chunks_below")
-    op.drop_column("persona", "llm_relevance_filter")
-    op.drop_column("persona", "llm_filter_extraction")
-    op.drop_column("persona", "recency_bias")
-
-
-def downgrade() -> None:
-    # Add back recency_bias column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "recency_bias",
-            sa.VARCHAR(),
-            nullable=False,
-            server_default="base_decay",
-        ),
-    )
-
-    # Add back llm_filter_extraction column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "llm_filter_extraction",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Add back llm_relevance_filter column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "llm_relevance_filter",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Add back chunks_below column
-    op.add_column(
-        "persona",
-        sa.Column("chunks_below", sa.Integer(), nullable=False, server_default="0"),
-    )
-
-    # Add back chunks_above column
-    op.add_column(
-        "persona",
-        sa.Column("chunks_above", sa.Integer(), nullable=False, server_default="0"),
-    )
-
-    # Add back num_chunks column
-    op.add_column("persona", sa.Column("num_chunks", sa.Float(), nullable=True))
-
-    # Add back is_default_persona column
-    op.add_column(
-        "persona",
-        sa.Column(
-            "is_default_persona",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.false(),
-        ),
-    )
-
-    # Migrate data from featured to is_default_persona
-    op.execute("UPDATE persona SET is_default_persona = featured")
-
-    # Drop featured column
-    op.drop_column("persona", "featured")

backend/alembic/versions/72aa7de2e5cf_make_processing_mode_default_all_caps.py

@@ -24,10 +24,12 @@ depends_on = None
 def upgrade() -> None:
     # Convert existing lowercase values to uppercase to match enum member names
     op.execute(
-        "UPDATE connector_credential_pair SET processing_mode = 'REGULAR' WHERE processing_mode = 'regular'"
+        "UPDATE connector_credential_pair SET processing_mode = 'REGULAR' "
+        "WHERE processing_mode = 'regular'"
     )
     op.execute(
-        "UPDATE connector_credential_pair SET processing_mode = 'FILE_SYSTEM' WHERE processing_mode = 'file_system'"
+        "UPDATE connector_credential_pair SET processing_mode = 'FILE_SYSTEM' "
+        "WHERE processing_mode = 'file_system'"
     )
 
     # Update the server default to use uppercase

backend/alembic/versions/7616121f6e97_add_enterprise_fields_to_scim_user_mapping.py

@@ -1,48 +0,0 @@
-"""add enterprise and name fields to scim_user_mapping
-
-Revision ID: 7616121f6e97
-Revises: 07b98176f1de
-Create Date: 2026-02-23 12:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "7616121f6e97"
-down_revision = "07b98176f1de"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("department", sa.String(), nullable=True),
-    )
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("manager", sa.String(), nullable=True),
-    )
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("given_name", sa.String(), nullable=True),
-    )
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("family_name", sa.String(), nullable=True),
-    )
-    op.add_column(
-        "scim_user_mapping",
-        sa.Column("scim_emails_json", sa.Text(), nullable=True),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("scim_user_mapping", "scim_emails_json")
-    op.drop_column("scim_user_mapping", "family_name")
-    op.drop_column("scim_user_mapping", "given_name")
-    op.drop_column("scim_user_mapping", "manager")
-    op.drop_column("scim_user_mapping", "department")

backend/alembic/versions/7b9b952abdf6_update_entities.py

@@ -289,7 +289,8 @@ def upgrade() -> None:
         attributes_str = json.dumps(attributes).replace("'", "''")
         op.execute(
             sa.text(
-                f"UPDATE kg_entity_type SET attributes = '{attributes_str}'WHERE id_name = '{entity_type}'"
+                f"UPDATE kg_entity_type SET attributes = '{attributes_str}'"
+                f"WHERE id_name = '{entity_type}'"
             ),
         )
 
@@ -311,6 +312,7 @@ def downgrade() -> None:
         attributes_str = json.dumps(attributes).replace("'", "''")
         op.execute(
             sa.text(
-                f"UPDATE kg_entity_type SET attributes = '{attributes_str}'WHERE id_name = '{entity_type}'"
+                f"UPDATE kg_entity_type SET attributes = '{attributes_str}'"
+                f"WHERE id_name = '{entity_type}'"
             ),
         )

backend/alembic/versions/7cb492013621_code_interpreter_server_model.py

@@ -1,31 +0,0 @@
-"""code interpreter server model
-
-Revision ID: 7cb492013621
-Revises: 0bb4558f35df
-Create Date: 2026-02-22 18:54:54.007265
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "7cb492013621"
-down_revision = "0bb4558f35df"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "code_interpreter_server",
-        sa.Column("id", sa.Integer, primary_key=True),
-        sa.Column(
-            "server_enabled", sa.Boolean, nullable=False, server_default=sa.true()
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_table("code_interpreter_server")

backend/alembic/versions/8ffcc2bcfc11_add_needs_persona_sync_to_user_file.py

@@ -1,33 +0,0 @@
-"""add needs_persona_sync to user_file
-
-Revision ID: 8ffcc2bcfc11
-Revises: 7616121f6e97
-Create Date: 2026-02-23 10:48:48.343826
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "8ffcc2bcfc11"
-down_revision = "7616121f6e97"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "user_file",
-        sa.Column(
-            "needs_persona_sync",
-            sa.Boolean(),
-            nullable=False,
-            server_default=sa.text("false"),
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("user_file", "needs_persona_sync")

backend/alembic/versions/90e3b9af7da4_tag_fix.py

@@ -160,7 +160,7 @@ def remove_old_tags() -> None:
                     f"""
                     DELETE FROM document__tag
                     WHERE document_id = '{document_id}'
-                    AND tag_id IN ({",".join(to_delete)})
+                    AND tag_id IN ({','.join(to_delete)})
                     """
                 )
             )
@@ -239,7 +239,7 @@ def _get_batch_documents_with_multiple_tags(
         ).fetchall()
         if not batch:
             break
-        doc_ids = [document_id for (document_id,) in batch]
+        doc_ids = [document_id for document_id, in batch]
         yield doc_ids
         offset_clause = f"AND document__tag.document_id > '{doc_ids[-1]}'"
 

backend/alembic/versions/93a2e195e25c_add_voice_provider_and_user_voice_prefs.py

@@ -1,117 +0,0 @@
-"""add_voice_provider_and_user_voice_prefs
-
-Revision ID: 93a2e195e25c
-Revises: 27fb147a843f
-Create Date: 2026-02-23 15:16:39.507304
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy import column
-from sqlalchemy import true
-from sqlalchemy.dialects import postgresql
-
-
-# revision identifiers, used by Alembic.
-revision = "93a2e195e25c"
-down_revision = "27fb147a843f"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Create voice_provider table
-    op.create_table(
-        "voice_provider",
-        sa.Column("id", sa.Integer(), primary_key=True),
-        sa.Column("name", sa.String(), unique=True, nullable=False),
-        sa.Column("provider_type", sa.String(), nullable=False),
-        sa.Column("api_key", sa.LargeBinary(), nullable=True),
-        sa.Column("api_base", sa.String(), nullable=True),
-        sa.Column("custom_config", postgresql.JSONB(), nullable=True),
-        sa.Column("stt_model", sa.String(), nullable=True),
-        sa.Column("tts_model", sa.String(), nullable=True),
-        sa.Column("default_voice", sa.String(), nullable=True),
-        sa.Column(
-            "is_default_stt", sa.Boolean(), nullable=False, server_default="false"
-        ),
-        sa.Column(
-            "is_default_tts", sa.Boolean(), nullable=False, server_default="false"
-        ),
-        sa.Column("deleted", sa.Boolean(), nullable=False, server_default="false"),
-        sa.Column(
-            "time_created",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            nullable=False,
-        ),
-        sa.Column(
-            "time_updated",
-            sa.DateTime(timezone=True),
-            server_default=sa.func.now(),
-            onupdate=sa.func.now(),
-            nullable=False,
-        ),
-    )
-
-    # Add partial unique indexes to enforce only one default STT/TTS provider
-    op.create_index(
-        "ix_voice_provider_one_default_stt",
-        "voice_provider",
-        ["is_default_stt"],
-        unique=True,
-        postgresql_where=column("is_default_stt") == true(),
-    )
-    op.create_index(
-        "ix_voice_provider_one_default_tts",
-        "voice_provider",
-        ["is_default_tts"],
-        unique=True,
-        postgresql_where=column("is_default_tts") == true(),
-    )
-
-    # Add voice preference columns to user table
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_auto_send",
-            sa.Boolean(),
-            default=False,
-            nullable=False,
-            server_default="false",
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_auto_playback",
-            sa.Boolean(),
-            default=False,
-            nullable=False,
-            server_default="false",
-        ),
-    )
-    op.add_column(
-        "user",
-        sa.Column(
-            "voice_playback_speed",
-            sa.Float(),
-            default=1.0,
-            nullable=False,
-            server_default="1.0",
-        ),
-    )
-
-
-def downgrade() -> None:
-    # Remove user voice preference columns
-    op.drop_column("user", "voice_playback_speed")
-    op.drop_column("user", "voice_auto_playback")
-    op.drop_column("user", "voice_auto_send")
-
-    op.drop_index("ix_voice_provider_one_default_tts", table_name="voice_provider")
-    op.drop_index("ix_voice_provider_one_default_stt", table_name="voice_provider")
-
-    # Drop voice_provider table
-    op.drop_table("voice_provider")

backend/alembic/versions/a01bf2971c5d_update_default_tool_descriptions.py

@@ -24,7 +24,8 @@ TOOL_DESCRIPTIONS = {
         "The action will be used when the user asks the agent to generate an image."
     ),
     "WebSearchTool": (
-        "The Web Search Action allows the agent to perform internet searches for up-to-date information."
+        "The Web Search Action allows the agent "
+        "to perform internet searches for up-to-date information."
     ),
     "KnowledgeGraphTool": (
         "The Knowledge Graph Search Action allows the agent to search the "

backend/alembic/versions/a3b8d9e2f1c4_make_scim_external_id_nullable.py

@@ -1,34 +0,0 @@
-"""make scim_user_mapping.external_id nullable
-
-Revision ID: a3b8d9e2f1c4
-Revises: 2664261bfaab
-Create Date: 2026-03-02
-
-"""
-
-from alembic import op
-
-
-# revision identifiers, used by Alembic.
-revision = "a3b8d9e2f1c4"
-down_revision = "2664261bfaab"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=True,
-    )
-
-
-def downgrade() -> None:
-    # Delete any rows where external_id is NULL before re-applying NOT NULL
-    op.execute("DELETE FROM scim_user_mapping WHERE external_id IS NULL")
-    op.alter_column(
-        "scim_user_mapping",
-        "external_id",
-        nullable=False,
-    )

backend/alembic/versions/b5c4d7e8f9a1_add_hierarchy_node_cc_pair_table.py

@@ -1,51 +0,0 @@
-"""add hierarchy_node_by_connector_credential_pair table
-
-Revision ID: b5c4d7e8f9a1
-Revises: a3b8d9e2f1c4
-Create Date: 2026-03-04
-
-"""
-
-import sqlalchemy as sa
-from alembic import op
-
-revision = "b5c4d7e8f9a1"
-down_revision = "a3b8d9e2f1c4"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.create_table(
-        "hierarchy_node_by_connector_credential_pair",
-        sa.Column("hierarchy_node_id", sa.Integer(), nullable=False),
-        sa.Column("connector_id", sa.Integer(), nullable=False),
-        sa.Column("credential_id", sa.Integer(), nullable=False),
-        sa.ForeignKeyConstraint(
-            ["hierarchy_node_id"],
-            ["hierarchy_node.id"],
-            ondelete="CASCADE",
-        ),
-        sa.ForeignKeyConstraint(
-            ["connector_id", "credential_id"],
-            [
-                "connector_credential_pair.connector_id",
-                "connector_credential_pair.credential_id",
-            ],
-            ondelete="CASCADE",
-        ),
-        sa.PrimaryKeyConstraint("hierarchy_node_id", "connector_id", "credential_id"),
-    )
-    op.create_index(
-        "ix_hierarchy_node_cc_pair_connector_credential",
-        "hierarchy_node_by_connector_credential_pair",
-        ["connector_id", "credential_id"],
-    )
-
-
-def downgrade() -> None:
-    op.drop_index(
-        "ix_hierarchy_node_cc_pair_connector_credential",
-        table_name="hierarchy_node_by_connector_credential_pair",
-    )
-    op.drop_table("hierarchy_node_by_connector_credential_pair")

backend/alembic/versions/c0c937d5c9e5_llm_provider_deprecate_fields.py

@@ -1,70 +0,0 @@
-"""llm provider deprecate fields
-
-Revision ID: c0c937d5c9e5
-Revises: 8ffcc2bcfc11
-Create Date: 2026-02-25 17:35:46.125102
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "c0c937d5c9e5"
-down_revision = "8ffcc2bcfc11"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Make default_model_name nullable (was NOT NULL)
-    op.alter_column(
-        "llm_provider",
-        "default_model_name",
-        existing_type=sa.String(),
-        nullable=True,
-    )
-
-    # Drop unique constraint on is_default_provider (defaults now tracked via LLMModelFlow)
-    op.drop_constraint(
-        "llm_provider_is_default_provider_key",
-        "llm_provider",
-        type_="unique",
-    )
-
-    # Remove server_default from is_default_vision_provider (was server_default=false())
-    op.alter_column(
-        "llm_provider",
-        "is_default_vision_provider",
-        existing_type=sa.Boolean(),
-        server_default=None,
-    )
-
-
-def downgrade() -> None:
-    # Restore default_model_name to NOT NULL (set empty string for any NULLs first)
-    op.execute(
-        "UPDATE llm_provider SET default_model_name = '' WHERE default_model_name IS NULL"
-    )
-    op.alter_column(
-        "llm_provider",
-        "default_model_name",
-        existing_type=sa.String(),
-        nullable=False,
-    )
-
-    # Restore unique constraint on is_default_provider
-    op.create_unique_constraint(
-        "llm_provider_is_default_provider_key",
-        "llm_provider",
-        ["is_default_provider"],
-    )
-
-    # Restore server_default for is_default_vision_provider
-    op.alter_column(
-        "llm_provider",
-        "is_default_vision_provider",
-        existing_type=sa.Boolean(),
-        server_default=sa.false(),
-    )

backend/alembic/versions/c7f2e1b4a9d3_add_sharing_scope_to_build_session.py

@@ -1,31 +0,0 @@
-"""add sharing_scope to build_session
-
-Revision ID: c7f2e1b4a9d3
-Revises: 19c0ccb01687
-Create Date: 2026-02-17 12:00:00.000000
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-revision = "c7f2e1b4a9d3"
-down_revision = "19c0ccb01687"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.add_column(
-        "build_session",
-        sa.Column(
-            "sharing_scope",
-            sa.String(),
-            nullable=False,
-            server_default="private",
-        ),
-    )
-
-
-def downgrade() -> None:
-    op.drop_column("build_session", "sharing_scope")

backend/alembic/versions/c9e2cd766c29_add_s3_file_store_table.py

@@ -140,7 +140,8 @@ def _migrate_files_to_postgres() -> None:
     # Fetch rows that have external storage pointers (bucket/object_key not NULL)
     result = session.execute(
         text(
-            "SELECT file_id, bucket_name, object_key FROM file_record WHERE bucket_name IS NOT NULL AND object_key IS NOT NULL"
+            "SELECT file_id, bucket_name, object_key FROM file_record "
+            "WHERE bucket_name IS NOT NULL AND object_key IS NOT NULL"
         )
     )
 
@@ -181,7 +182,8 @@ def _migrate_files_to_postgres() -> None:
             # Update DB row: set lobj_oid, clear bucket/object_key
             session.execute(
                 text(
-                    "UPDATE file_record SET lobj_oid = :lobj_oid, bucket_name = NULL, object_key = NULL WHERE file_id = :file_id"
+                    "UPDATE file_record SET lobj_oid = :lobj_oid, bucket_name = NULL, "
+                    "object_key = NULL WHERE file_id = :file_id"
                 ),
                 {"lobj_oid": lobj_oid, "file_id": file_id},
             )
@@ -222,7 +224,8 @@ def _migrate_files_to_external_storage() -> None:
     # Find all files currently stored in PostgreSQL (lobj_oid is not null)
     result = session.execute(
         text(
-            "SELECT file_id FROM file_record WHERE lobj_oid IS NOT NULL AND bucket_name IS NULL AND object_key IS NULL"
+            "SELECT file_id FROM file_record WHERE lobj_oid IS NOT NULL "
+            "AND bucket_name IS NULL AND object_key IS NULL"
         )
     )
 

backend/alembic/versions/d09fc20a3c66_seed_builtin_tools.py

@@ -39,7 +39,8 @@ BUILT_IN_TOOLS = [
         "name": "WebSearchTool",
         "display_name": "Web Search",
         "description": (
-            "The Web Search Action allows the assistant to perform internet searches for up-to-date information."
+            "The Web Search Action allows the assistant "
+            "to perform internet searches for up-to-date information."
         ),
         "in_code_tool_id": "WebSearchTool",
     },

backend/alembic_tenants/versions/3b9f09038764_add_read_only_kg_user.py

@@ -11,6 +11,7 @@ from sqlalchemy import text
 from alembic import op
 from onyx.configs.app_configs import DB_READONLY_PASSWORD
 from onyx.configs.app_configs import DB_READONLY_USER
+from shared_configs.configs import MULTI_TENANT
 
 
 # revision identifiers, used by Alembic.
@@ -21,52 +22,59 @@ depends_on = None
 
 
 def upgrade() -> None:
-    # Enable pg_trgm extension if not already enabled
-    op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
+    if MULTI_TENANT:
 
-    # Create the read-only db user if it does not already exist.
-    if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
-        raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+        # Enable pg_trgm extension if not already enabled
+        op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
 
-    op.execute(
-        text(
-            f"""
-            DO $$
-            BEGIN
-                -- Check if the read-only user already exists
-                IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                    -- Create the read-only user with the specified password
-                    EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
-                    -- First revoke all privileges to ensure a clean slate
-                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                    -- Grant only the CONNECT privilege to allow the user to connect to the database
-                    -- but not perform any operations without additional specific grants
-                    EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
-                END IF;
-            END
-            $$;
-            """
+        # Create read-only db user here only in multi-tenant mode. For single-tenant mode,
+        # the user is created in the standard migration.
+        if not (DB_READONLY_USER and DB_READONLY_PASSWORD):
+            raise Exception("DB_READONLY_USER or DB_READONLY_PASSWORD is not set")
+
+        op.execute(
+            text(
+                f"""
+                DO $$
+                BEGIN
+                    -- Check if the read-only user already exists
+                    IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                        -- Create the read-only user with the specified password
+                        EXECUTE format('CREATE USER %I WITH PASSWORD %L', '{DB_READONLY_USER}', '{DB_READONLY_PASSWORD}');
+                        -- First revoke all privileges to ensure a clean slate
+                        EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                        -- Grant only the CONNECT privilege to allow the user to connect to the database
+                        -- but not perform any operations without additional specific grants
+                        EXECUTE format('GRANT CONNECT ON DATABASE %I TO %I', current_database(), '{DB_READONLY_USER}');
+                    END IF;
+                END
+                $$;
+                """
+            )
         )
-    )
 
 
 def downgrade() -> None:
-    op.execute(
-        text(
-            f"""
-        DO $$
-        BEGIN
-            IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
-                -- First revoke all privileges from the database
-                EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
-                -- Then revoke all privileges from the public schema
-                EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
-                -- Then drop the user
-                EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
-            END IF;
-        END
-        $$;
-    """
+    if MULTI_TENANT:
+        # Drop read-only db user here only in single tenant mode. For multi-tenant mode,
+        # the user is dropped in the alembic_tenants migration.
+
+        op.execute(
+            text(
+                f"""
+            DO $$
+            BEGIN
+                IF EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '{DB_READONLY_USER}') THEN
+                    -- First revoke all privileges from the database
+                    EXECUTE format('REVOKE ALL ON DATABASE %I FROM %I', current_database(), '{DB_READONLY_USER}');
+                    -- Then revoke all privileges from the public schema
+                    EXECUTE format('REVOKE ALL ON SCHEMA public FROM %I', '{DB_READONLY_USER}');
+                    -- Then drop the user
+                    EXECUTE format('DROP USER %I', '{DB_READONLY_USER}');
+                END IF;
+            END
+            $$;
+        """
+            )
         )
-    )
-    op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))
+        op.execute(text("DROP EXTENSION IF EXISTS pg_trgm"))

backend/ee/onyx/access/access.py

@@ -9,15 +9,12 @@ from onyx.access.access import (
     _get_access_for_documents as get_access_for_documents_without_groups,
 )
 from onyx.access.access import _get_acl_for_user as get_acl_for_user_without_groups
-from onyx.access.access import collect_user_file_access
 from onyx.access.models import DocumentAccess
 from onyx.access.utils import prefix_external_group
 from onyx.access.utils import prefix_user_group
 from onyx.db.document import get_document_sources
 from onyx.db.document import get_documents_by_ids
 from onyx.db.models import User
-from onyx.db.models import UserFile
-from onyx.db.user_file import fetch_user_files_with_access_relationships
 from onyx.utils.logger import setup_logger
 
 
@@ -119,68 +116,6 @@ def _get_access_for_documents(
     return access_map
 
 
-def _collect_user_file_group_names(user_file: UserFile) -> set[str]:
-    """Extract user-group names from the already-loaded Persona.groups
-    relationships on a UserFile (skipping deleted personas)."""
-    groups: set[str] = set()
-    for persona in user_file.assistants:
-        if persona.deleted:
-            continue
-        for group in persona.groups:
-            groups.add(group.name)
-    return groups
-
-
-def get_access_for_user_files_impl(
-    user_file_ids: list[str],
-    db_session: Session,
-) -> dict[str, DocumentAccess]:
-    """EE version: extends the MIT user file ACL with user group names
-    from personas shared via user groups.
-
-    Uses a single DB query (via fetch_user_files_with_access_relationships)
-    that eagerly loads both the MIT-needed and EE-needed relationships.
-
-    NOTE: is imported in onyx.access.access by `fetch_versioned_implementation`
-    DO NOT REMOVE."""
-    user_files = fetch_user_files_with_access_relationships(
-        user_file_ids, db_session, eager_load_groups=True
-    )
-    return build_access_for_user_files_impl(user_files)
-
-
-def build_access_for_user_files_impl(
-    user_files: list[UserFile],
-) -> dict[str, DocumentAccess]:
-    """EE version: works on pre-loaded UserFile objects.
-    Expects Persona.groups to be eagerly loaded.
-
-    NOTE: is imported in onyx.access.access by `fetch_versioned_implementation`
-    DO NOT REMOVE."""
-    result: dict[str, DocumentAccess] = {}
-    for user_file in user_files:
-        if user_file.user is None:
-            result[str(user_file.id)] = DocumentAccess.build(
-                user_emails=[],
-                user_groups=[],
-                is_public=True,
-                external_user_emails=[],
-                external_user_group_ids=[],
-            )
-            continue
-
-        emails, is_public = collect_user_file_access(user_file)
-        group_names = _collect_user_file_group_names(user_file)
-        result[str(user_file.id)] = DocumentAccess.build(
-            user_emails=list(emails),
-            user_groups=list(group_names),
-            is_public=is_public,
-            external_user_emails=[],
-            external_user_group_ids=[],
-        )
-    return result
-
-
 def _get_acl_for_user(user: User, db_session: Session) -> set[str]:
     """Returns a list of ACL entries that the user has access to. This is meant to be
     used downstream to filter out documents that the user does not have access to. The

backend/ee/onyx/auth/users.py

@@ -1,4 +1,3 @@
-import os
 from datetime import datetime
 
 import jwt
@@ -21,12 +20,7 @@ logger = setup_logger()
 
 
 def verify_auth_setting() -> None:
-    # All the Auth flows are valid for EE version, but warn about deprecated 'disabled'
-    raw_auth_type = (os.environ.get("AUTH_TYPE") or "").lower()
-    if raw_auth_type == "disabled":
-        logger.warning(
-            "AUTH_TYPE='disabled' is no longer supported. Using 'basic' instead. Please update your configuration."
-        )
+    # All the Auth flows are valid for EE version
     logger.notice(f"Using Auth Type: {AUTH_TYPE.value}")
 
 

backend/ee/onyx/background/celery/apps/background.py

@@ -0,0 +1,15 @@
+from onyx.background.celery.apps import app_base
+from onyx.background.celery.apps.background import celery_app
+
+
+celery_app.autodiscover_tasks(
+    app_base.filter_task_modules(
+        [
+            "ee.onyx.background.celery.tasks.doc_permission_syncing",
+            "ee.onyx.background.celery.tasks.external_group_syncing",
+            "ee.onyx.background.celery.tasks.cleanup",
+            "ee.onyx.background.celery.tasks.tenant_provisioning",
+            "ee.onyx.background.celery.tasks.query_history",
+        ]
+    )
+)

backend/ee/onyx/background/celery/tasks/cloud/tasks.py

@@ -59,6 +59,7 @@ def cloud_beat_task_generator(
         # gated_tenants = get_gated_tenants()
 
         for tenant_id in tenant_ids:
+
             # Same comment here as the above NOTE
             # if tenant_id in gated_tenants:
             #     continue

backend/ee/onyx/background/celery/tasks/doc_permission_syncing/tasks.py

@@ -424,7 +424,10 @@ def connector_permission_sync_generator_task(
             raise ValueError(error_msg)
 
         if not redis_connector.permissions.fenced:  # The fence must exist
-            error_msg = f"connector_permission_sync_generator_task - fence not found: fence={redis_connector.permissions.fence_key}"
+            error_msg = (
+                f"connector_permission_sync_generator_task - fence not found: "
+                f"fence={redis_connector.permissions.fence_key}"
+            )
             _fail_doc_permission_sync_attempt(attempt_id, error_msg)
             raise ValueError(error_msg)
 
@@ -438,7 +441,8 @@ def connector_permission_sync_generator_task(
 
         if payload.celery_task_id is None:
             logger.info(
-                f"connector_permission_sync_generator_task - Waiting for fence: fence={redis_connector.permissions.fence_key}"
+                f"connector_permission_sync_generator_task - Waiting for fence: "
+                f"fence={redis_connector.permissions.fence_key}"
             )
             sleep(1)
             continue
@@ -604,7 +608,8 @@ def connector_permission_sync_generator_task(
                 docs_with_permission_errors=docs_with_errors,
             )
             task_logger.info(
-                f"Completed doc permission sync attempt {attempt_id}: {tasks_generated} docs, {docs_with_errors} errors"
+                f"Completed doc permission sync attempt {attempt_id}: "
+                f"{tasks_generated} docs, {docs_with_errors} errors"
             )
 
             redis_connector.permissions.generator_complete = tasks_generated
@@ -711,7 +716,9 @@ def element_update_permissions(
 
             elapsed = time.monotonic() - start
             task_logger.info(
-                f"{element_type}={element_id} action=update_permissions elapsed={elapsed:.2f}"
+                f"{element_type}={element_id} "
+                f"action=update_permissions "
+                f"elapsed={elapsed:.2f}"
             )
     except Exception as e:
         task_logger.exception(
@@ -893,7 +900,8 @@ def validate_permission_sync_fence(
         tasks_not_in_celery += 1
 
     task_logger.info(
-        f"validate_permission_sync_fence task check: tasks_scanned={tasks_scanned} tasks_not_in_celery={tasks_not_in_celery}"
+        "validate_permission_sync_fence task check: "
+        f"tasks_scanned={tasks_scanned} tasks_not_in_celery={tasks_not_in_celery}"
     )
 
     # we're active if there are still tasks to run and those tasks all exist in celery
@@ -999,10 +1007,7 @@ class PermissionSyncCallback(IndexingHeartbeatInterface):
 
 
 def monitor_ccpair_permissions_taskset(
-    tenant_id: str,
-    key_bytes: bytes,
-    r: Redis,  # noqa: ARG001
-    db_session: Session,
+    tenant_id: str, key_bytes: bytes, r: Redis, db_session: Session  # noqa: ARG001
 ) -> None:
     fence_key = key_bytes.decode("utf-8")
     cc_pair_id_str = RedisConnector.get_id_from_fence_key(fence_key)
@@ -1026,7 +1031,8 @@ def monitor_ccpair_permissions_taskset(
         payload = redis_connector.permissions.payload
     except ValidationError:
         task_logger.exception(
-            "Permissions sync payload failed to validate. Schema may have been updated."
+            "Permissions sync payload failed to validate. "
+            "Schema may have been updated."
         )
         return
 
@@ -1035,7 +1041,11 @@ def monitor_ccpair_permissions_taskset(
 
     remaining = redis_connector.permissions.get_remaining()
     task_logger.info(
-        f"Permissions sync progress: cc_pair={cc_pair_id} id={payload.id} remaining={remaining} initial={initial}"
+        f"Permissions sync progress: "
+        f"cc_pair={cc_pair_id} "
+        f"id={payload.id} "
+        f"remaining={remaining} "
+        f"initial={initial}"
     )
 
     # Add telemetry for permission syncing progress
@@ -1054,7 +1064,10 @@ def monitor_ccpair_permissions_taskset(
 
     mark_cc_pair_as_permissions_synced(db_session, int(cc_pair_id), payload.started)
     task_logger.info(
-        f"Permissions sync finished: cc_pair={cc_pair_id} id={payload.id} num_synced={initial}"
+        f"Permissions sync finished: "
+        f"cc_pair={cc_pair_id} "
+        f"id={payload.id} "
+        f"num_synced={initial}"
     )
 
     # Add telemetry for permission syncing complete

backend/ee/onyx/background/celery/tasks/external_group_syncing/tasks.py

@@ -111,20 +111,23 @@ def _is_external_group_sync_due(cc_pair: ConnectorCredentialPair) -> bool:
 
     if cc_pair.access_type != AccessType.SYNC:
         task_logger.error(
-            f"Received non-sync CC Pair {cc_pair.id} for external group sync. Actual access type: {cc_pair.access_type}"
+            f"Received non-sync CC Pair {cc_pair.id} for external "
+            f"group sync. Actual access type: {cc_pair.access_type}"
         )
         return False
 
     if cc_pair.status == ConnectorCredentialPairStatus.DELETING:
         task_logger.debug(
-            f"Skipping group sync for CC Pair {cc_pair.id} - CC Pair is being deleted"
+            f"Skipping group sync for CC Pair {cc_pair.id} - "
+            f"CC Pair is being deleted"
         )
         return False
 
     sync_config = get_source_perm_sync_config(cc_pair.connector.source)
     if sync_config is None:
         task_logger.debug(
-            f"Skipping group sync for CC Pair {cc_pair.id} - no sync config found for {cc_pair.connector.source}"
+            f"Skipping group sync for CC Pair {cc_pair.id} - "
+            f"no sync config found for {cc_pair.connector.source}"
         )
         return False
 
@@ -132,7 +135,8 @@ def _is_external_group_sync_due(cc_pair: ConnectorCredentialPair) -> bool:
     # This is fine because all sources dont necessarily have a concept of groups
     if sync_config.group_sync_config is None:
         task_logger.debug(
-            f"Skipping group sync for CC Pair {cc_pair.id} - no group sync config found for {cc_pair.connector.source}"
+            f"Skipping group sync for CC Pair {cc_pair.id} - "
+            f"no group sync config found for {cc_pair.connector.source}"
         )
         return False
 

backend/ee/onyx/background/celery/tasks/ttl_management/tasks.py

@@ -74,7 +74,8 @@ def perform_ttl_management_task(
 
     except Exception:
         logger.exception(
-            f"delete_chat_session exceptioned. user_id={user_id} session_id={session_id}"
+            "delete_chat_session exceptioned. "
+            f"user_id={user_id} session_id={session_id}"
         )
         with get_session_with_current_tenant() as db_session:
             mark_task_as_finished_with_id(

backend/ee/onyx/background/task_name_builders.py

@@ -7,8 +7,7 @@ QUERY_HISTORY_TASK_NAME_PREFIX = OnyxCeleryTask.EXPORT_QUERY_HISTORY_TASK
 
 
 def name_chat_ttl_task(
-    retention_limit_days: float,
-    tenant_id: str | None = None,  # noqa: ARG001
+    retention_limit_days: float, tenant_id: str | None = None  # noqa: ARG001
 ) -> str:
     return f"chat_ttl_{retention_limit_days}_days"
 

backend/ee/onyx/db/analytics.py

@@ -31,8 +31,7 @@ def fetch_query_analytics(
             func.sum(case((ChatMessageFeedback.is_positive, 1), else_=0)),
             func.sum(
                 case(
-                    (ChatMessageFeedback.is_positive == False, 1),  # noqa: E712
-                    else_=0,  # noqa: E712
+                    (ChatMessageFeedback.is_positive == False, 1), else_=0  # noqa: E712
                 )
             ),
             cast(ChatMessage.time_sent, Date),
@@ -67,8 +66,7 @@ def fetch_per_user_query_analytics(
             func.sum(case((ChatMessageFeedback.is_positive, 1), else_=0)),
             func.sum(
                 case(
-                    (ChatMessageFeedback.is_positive == False, 1),  # noqa: E712
-                    else_=0,  # noqa: E712
+                    (ChatMessageFeedback.is_positive == False, 1), else_=0  # noqa: E712
                 )
             ),
             cast(ChatMessage.time_sent, Date),

backend/ee/onyx/db/connector_credential_pair.py

@@ -23,7 +23,8 @@ def _delete_connector_credential_pair_user_groups_relationship__no_commit(
     )
     if cc_pair is None:
         raise ValueError(
-            f"ConnectorCredentialPair with connector_id: {connector_id} and credential_id: {credential_id} not found"
+            f"ConnectorCredentialPair with connector_id: {connector_id} "
+            f"and credential_id: {credential_id} not found"
         )
 
     stmt = delete(UserGroup__ConnectorCredentialPair).where(

backend/ee/onyx/db/external_perm.py

@@ -123,7 +123,8 @@ def upsert_external_groups(
             user_id = email_id_map.get(user_email.lower())
             if user_id is None:
                 logger.warning(
-                    f"User in group {external_group.id} with email {user_email} not found"
+                    f"User in group {external_group.id}"
+                    f" with email {user_email} not found"
                 )
                 continue
 

backend/ee/onyx/db/hierarchy.py

@@ -18,7 +18,7 @@ from onyx.db.models import HierarchyNode
 
 
 def _build_hierarchy_access_filter(
-    user_email: str,
+    user_email: str | None,
     external_group_ids: list[str],
 ) -> ColumnElement[bool]:
     """Build SQLAlchemy filter for hierarchy node access.
@@ -43,7 +43,7 @@ def _build_hierarchy_access_filter(
 def _get_accessible_hierarchy_nodes_for_source(
     db_session: Session,
     source: DocumentSource,
-    user_email: str,
+    user_email: str | None,
     external_group_ids: list[str],
 ) -> list[HierarchyNode]:
     """

backend/ee/onyx/db/license.py

@@ -11,10 +11,11 @@ from ee.onyx.server.license.models import LicenseMetadata
 from ee.onyx.server.license.models import LicensePayload
 from ee.onyx.server.license.models import LicenseSource
 from onyx.auth.schemas import UserRole
-from onyx.cache.factory import get_cache_backend
 from onyx.configs.constants import ANONYMOUS_USER_EMAIL
 from onyx.db.models import License
 from onyx.db.models import User
+from onyx.redis.redis_pool import get_redis_client
+from onyx.redis.redis_pool import get_redis_replica_client
 from onyx.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.contextvars import get_current_tenant_id
@@ -141,7 +142,7 @@ def get_used_seats(tenant_id: str | None = None) -> int:
 
 def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata | None:
     """
-    Get license metadata from cache.
+    Get license metadata from Redis cache.
 
     Args:
         tenant_id: Tenant ID (for multi-tenant deployments)
@@ -149,34 +150,38 @@ def get_cached_license_metadata(tenant_id: str | None = None) -> LicenseMetadata
     Returns:
         LicenseMetadata if cached, None otherwise
     """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cached = cache.get(LICENSE_METADATA_KEY)
-    if not cached:
-        return None
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_replica_client(tenant_id=tenant)
 
-    try:
-        cached_str = (
-            cached.decode("utf-8") if isinstance(cached, bytes) else str(cached)
-        )
-        return LicenseMetadata.model_validate_json(cached_str)
-    except Exception as e:
-        logger.warning(f"Failed to parse cached license metadata: {e}")
-        return None
+    cached = redis_client.get(LICENSE_METADATA_KEY)
+    if cached:
+        try:
+            cached_str: str
+            if isinstance(cached, bytes):
+                cached_str = cached.decode("utf-8")
+            else:
+                cached_str = str(cached)
+            return LicenseMetadata.model_validate_json(cached_str)
+        except Exception as e:
+            logger.warning(f"Failed to parse cached license metadata: {e}")
+            return None
+    return None
 
 
 def invalidate_license_cache(tenant_id: str | None = None) -> None:
     """
     Invalidate the license metadata cache (not the license itself).
 
-    Deletes the cached LicenseMetadata. The actual license in the database
-    is not affected. Delete is idempotent — if the key doesn't exist, this
-    is a no-op.
+    This deletes the cached LicenseMetadata from Redis. The actual license
+    in the database is not affected. Redis delete is idempotent - if the
+    key doesn't exist, this is a no-op.
 
     Args:
         tenant_id: Tenant ID (for multi-tenant deployments)
     """
-    cache = get_cache_backend(tenant_id=tenant_id)
-    cache.delete(LICENSE_METADATA_KEY)
+    tenant = tenant_id or get_current_tenant_id()
+    redis_client = get_redis_client(tenant_id=tenant)
+    redis_client.delete(LICENSE_METADATA_KEY)
     logger.info("License cache invalidated")
 
 
@@ -187,7 +192,7 @@ def update_license_cache(
     tenant_id: str | None = None,
 ) -> LicenseMetadata:
     """
-    Update the cache with license metadata.
+    Update the Redis cache with license metadata.
 
     We cache all license statuses (ACTIVE, GRACE_PERIOD, GATED_ACCESS) because:
     1. Frontend needs status to show appropriate UI/banners
@@ -206,7 +211,7 @@ def update_license_cache(
     from ee.onyx.utils.license import get_license_status
 
     tenant = tenant_id or get_current_tenant_id()
-    cache = get_cache_backend(tenant_id=tenant_id)
+    redis_client = get_redis_client(tenant_id=tenant)
 
     used_seats = get_used_seats(tenant)
     status = get_license_status(payload, grace_period_end)
@@ -225,7 +230,7 @@ def update_license_cache(
         stripe_subscription_id=payload.stripe_subscription_id,
     )
 
-    cache.set(
+    redis_client.set(
         LICENSE_METADATA_KEY,
         metadata.model_dump_json(),
         ex=LICENSE_CACHE_TTL_SECONDS,
@@ -258,15 +263,9 @@ def refresh_license_cache(
 
     try:
         payload = verify_license_signature(license_record.license_data)
-        # Derive source from payload: manual licenses lack stripe_customer_id
-        source: LicenseSource = (
-            LicenseSource.AUTO_FETCH
-            if payload.stripe_customer_id
-            else LicenseSource.MANUAL_UPLOAD
-        )
         return update_license_cache(
             payload,
-            source=source,
+            source=LicenseSource.AUTO_FETCH,
             tenant_id=tenant_id,
         )
     except ValueError as e:

backend/ee/onyx/db/persona.py

@@ -7,7 +7,6 @@ from onyx.db.models import Persona
 from onyx.db.models import Persona__User
 from onyx.db.models import Persona__UserGroup
 from onyx.db.notification import create_notification
-from onyx.db.persona import mark_persona_user_files_for_sync
 from onyx.server.features.persona.models import PersonaSharedNotificationData
 
 
@@ -27,9 +26,7 @@ def update_persona_access(
 
     NOTE: Callers are responsible for committing."""
 
-    needs_sync = False
     if is_public is not None:
-        needs_sync = True
         persona = db_session.query(Persona).filter(Persona.id == persona_id).first()
         if persona:
             persona.is_public = is_public
@@ -38,7 +35,6 @@ def update_persona_access(
     # and a non-empty list means "replace with these shares".
 
     if user_ids is not None:
-        needs_sync = True
         db_session.query(Persona__User).filter(
             Persona__User.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -58,7 +54,6 @@ def update_persona_access(
                 )
 
     if group_ids is not None:
-        needs_sync = True
         db_session.query(Persona__UserGroup).filter(
             Persona__UserGroup.persona_id == persona_id
         ).delete(synchronize_session="fetch")
@@ -68,7 +63,3 @@ def update_persona_access(
             db_session.add(
                 Persona__UserGroup(persona_id=persona_id, user_group_id=group_id)
             )
-
-    # When sharing changes, user file ACLs need to be updated in the vector DB
-    if needs_sync:
-        mark_persona_user_files_for_sync(persona_id, db_session)

backend/ee/onyx/db/scim.py

@@ -1,721 +0,0 @@
-"""SCIM Data Access Layer.
-
-All database operations for SCIM provisioning — token management, user
-mappings, and group mappings. Extends the base DAL (see ``onyx.db.dal``).
-
-Usage from FastAPI::
-
-    def get_scim_dal(db_session: Session = Depends(get_session)) -> ScimDAL:
-        return ScimDAL(db_session)
-
-    @router.post("/tokens")
-    def create_token(dal: ScimDAL = Depends(get_scim_dal)) -> ...:
-        token = dal.create_token(name=..., hashed_token=..., ...)
-        dal.commit()
-        return token
-
-Usage from background tasks::
-
-    with ScimDAL.from_tenant("tenant_abc") as dal:
-        mapping = dal.create_user_mapping(external_id="idp-123", user_id=uid)
-        dal.commit()
-"""
-
-from __future__ import annotations
-
-from uuid import UUID
-
-from sqlalchemy import delete as sa_delete
-from sqlalchemy import func
-from sqlalchemy import Select
-from sqlalchemy import select
-from sqlalchemy import SQLColumnExpression
-from sqlalchemy.dialects.postgresql import insert as pg_insert
-
-from ee.onyx.server.scim.filtering import ScimFilter
-from ee.onyx.server.scim.filtering import ScimFilterOperator
-from ee.onyx.server.scim.models import ScimMappingFields
-from onyx.db.dal import DAL
-from onyx.db.models import ScimGroupMapping
-from onyx.db.models import ScimToken
-from onyx.db.models import ScimUserMapping
-from onyx.db.models import User
-from onyx.db.models import User__UserGroup
-from onyx.db.models import UserGroup
-from onyx.db.models import UserRole
-from onyx.utils.logger import setup_logger
-
-logger = setup_logger()
-
-
-class ScimDAL(DAL):
-    """Data Access Layer for SCIM provisioning operations.
-
-    Methods mutate but do NOT commit — call ``dal.commit()`` explicitly
-    when you want to persist changes. This follows the existing ``_no_commit``
-    convention and lets callers batch multiple operations into one transaction.
-    """
-
-    # ------------------------------------------------------------------
-    # Token operations
-    # ------------------------------------------------------------------
-
-    def create_token(
-        self,
-        name: str,
-        hashed_token: str,
-        token_display: str,
-        created_by_id: UUID,
-    ) -> ScimToken:
-        """Create a new SCIM bearer token.
-
-        Only one token is active at a time — this method automatically revokes
-        all existing active tokens before creating the new one.
-        """
-        # Revoke any currently active tokens
-        active_tokens = list(
-            self._session.scalars(
-                select(ScimToken).where(ScimToken.is_active.is_(True))
-            ).all()
-        )
-        for t in active_tokens:
-            t.is_active = False
-
-        token = ScimToken(
-            name=name,
-            hashed_token=hashed_token,
-            token_display=token_display,
-            created_by_id=created_by_id,
-        )
-        self._session.add(token)
-        self._session.flush()
-        return token
-
-    def get_active_token(self) -> ScimToken | None:
-        """Return the single currently active token, or None."""
-        return self._session.scalar(
-            select(ScimToken).where(ScimToken.is_active.is_(True))
-        )
-
-    def get_token_by_hash(self, hashed_token: str) -> ScimToken | None:
-        """Look up a token by its SHA-256 hash."""
-        return self._session.scalar(
-            select(ScimToken).where(ScimToken.hashed_token == hashed_token)
-        )
-
-    def revoke_token(self, token_id: int) -> None:
-        """Deactivate a token by ID.
-
-        Raises:
-            ValueError: If the token does not exist.
-        """
-        token = self._session.get(ScimToken, token_id)
-        if not token:
-            raise ValueError(f"SCIM token with id {token_id} not found")
-        token.is_active = False
-
-    def update_token_last_used(self, token_id: int) -> None:
-        """Update the last_used_at timestamp for a token."""
-        token = self._session.get(ScimToken, token_id)
-        if token:
-            token.last_used_at = func.now()  # type: ignore[assignment]
-
-    # ------------------------------------------------------------------
-    # User mapping operations
-    # ------------------------------------------------------------------
-
-    def create_user_mapping(
-        self,
-        external_id: str | None,
-        user_id: UUID,
-        scim_username: str | None = None,
-        fields: ScimMappingFields | None = None,
-    ) -> ScimUserMapping:
-        """Create a SCIM mapping for a user.
-
-        ``external_id`` may be ``None`` when the IdP omits it (RFC 7643
-        allows this). The mapping still marks the user as SCIM-managed.
-        """
-        f = fields or ScimMappingFields()
-        mapping = ScimUserMapping(
-            external_id=external_id,
-            user_id=user_id,
-            scim_username=scim_username,
-            department=f.department,
-            manager=f.manager,
-            given_name=f.given_name,
-            family_name=f.family_name,
-            scim_emails_json=f.scim_emails_json,
-        )
-        self._session.add(mapping)
-        self._session.flush()
-        return mapping
-
-    def get_user_mapping_by_external_id(
-        self, external_id: str
-    ) -> ScimUserMapping | None:
-        """Look up a user mapping by the IdP's external identifier."""
-        return self._session.scalar(
-            select(ScimUserMapping).where(ScimUserMapping.external_id == external_id)
-        )
-
-    def get_user_mapping_by_user_id(self, user_id: UUID) -> ScimUserMapping | None:
-        """Look up a user mapping by the Onyx user ID."""
-        return self._session.scalar(
-            select(ScimUserMapping).where(ScimUserMapping.user_id == user_id)
-        )
-
-    def list_user_mappings(
-        self,
-        start_index: int = 1,
-        count: int = 100,
-    ) -> tuple[list[ScimUserMapping], int]:
-        """List user mappings with SCIM-style pagination.
-
-        Args:
-            start_index: 1-based start index (SCIM convention).
-            count: Maximum number of results to return.
-
-        Returns:
-            A tuple of (mappings, total_count).
-        """
-        total = (
-            self._session.scalar(select(func.count()).select_from(ScimUserMapping)) or 0
-        )
-
-        offset = max(start_index - 1, 0)
-        mappings = list(
-            self._session.scalars(
-                select(ScimUserMapping)
-                .order_by(ScimUserMapping.id)
-                .offset(offset)
-                .limit(count)
-            ).all()
-        )
-
-        return mappings, total
-
-    def update_user_mapping_external_id(
-        self,
-        mapping_id: int,
-        external_id: str,
-    ) -> ScimUserMapping:
-        """Update the external ID on a user mapping.
-
-        Raises:
-            ValueError: If the mapping does not exist.
-        """
-        mapping = self._session.get(ScimUserMapping, mapping_id)
-        if not mapping:
-            raise ValueError(f"SCIM user mapping with id {mapping_id} not found")
-        mapping.external_id = external_id
-        return mapping
-
-    def delete_user_mapping(self, mapping_id: int) -> None:
-        """Delete a user mapping by ID. No-op if already deleted."""
-        mapping = self._session.get(ScimUserMapping, mapping_id)
-        if not mapping:
-            logger.warning("SCIM user mapping %d not found during delete", mapping_id)
-            return
-        self._session.delete(mapping)
-
-    # ------------------------------------------------------------------
-    # User query operations
-    # ------------------------------------------------------------------
-
-    def get_user(self, user_id: UUID) -> User | None:
-        """Fetch a user by ID."""
-        return self._session.scalar(
-            select(User).where(User.id == user_id)  # type: ignore[arg-type]
-        )
-
-    def get_user_by_email(self, email: str) -> User | None:
-        """Fetch a user by email (case-insensitive)."""
-        return self._session.scalar(
-            select(User).where(func.lower(User.email) == func.lower(email))
-        )
-
-    def add_user(self, user: User) -> None:
-        """Add a new user to the session and flush to assign an ID."""
-        self._session.add(user)
-        self._session.flush()
-
-    def update_user(
-        self,
-        user: User,
-        *,
-        email: str | None = None,
-        is_active: bool | None = None,
-        personal_name: str | None = None,
-    ) -> None:
-        """Update user attributes. Only sets fields that are provided."""
-        if email is not None:
-            user.email = email
-        if is_active is not None:
-            user.is_active = is_active
-        if personal_name is not None:
-            user.personal_name = personal_name
-
-    def deactivate_user(self, user: User) -> None:
-        """Mark a user as inactive."""
-        user.is_active = False
-
-    def list_users(
-        self,
-        scim_filter: ScimFilter | None,
-        start_index: int = 1,
-        count: int = 100,
-    ) -> tuple[list[tuple[User, ScimUserMapping | None]], int]:
-        """Query users with optional SCIM filter and pagination.
-
-        Returns:
-            A tuple of (list of (user, mapping) pairs, total_count).
-
-        Raises:
-            ValueError: If the filter uses an unsupported attribute.
-        """
-        # Inner-join with ScimUserMapping so only SCIM-managed users appear.
-        # Pre-existing system accounts (anonymous, admin, etc.) are excluded
-        # unless they were explicitly linked via SCIM provisioning.
-        query = (
-            select(User)
-            .join(ScimUserMapping, ScimUserMapping.user_id == User.id)
-            .where(User.role.notin_([UserRole.SLACK_USER, UserRole.EXT_PERM_USER]))
-        )
-
-        if scim_filter:
-            attr = scim_filter.attribute.lower()
-            if attr == "username":
-                # arg-type: fastapi-users types User.email as str, not a column expression
-                # assignment: union return type widens but query is still Select[tuple[User]]
-                query = _apply_scim_string_op(query, User.email, scim_filter)  # type: ignore[arg-type, assignment]
-            elif attr == "active":
-                query = query.where(
-                    User.is_active.is_(scim_filter.value.lower() == "true")  # type: ignore[attr-defined]
-                )
-            elif attr == "externalid":
-                mapping = self.get_user_mapping_by_external_id(scim_filter.value)
-                if not mapping:
-                    return [], 0
-                query = query.where(User.id == mapping.user_id)  # type: ignore[arg-type]
-            else:
-                raise ValueError(
-                    f"Unsupported filter attribute: {scim_filter.attribute}"
-                )
-
-        # Count total matching rows first, then paginate. SCIM uses 1-based
-        # indexing (RFC 7644 §3.4.2), so we convert to a 0-based offset.
-        total = (
-            self._session.scalar(select(func.count()).select_from(query.subquery()))
-            or 0
-        )
-
-        offset = max(start_index - 1, 0)
-        users = list(
-            self._session.scalars(
-                query.order_by(User.id).offset(offset).limit(count)  # type: ignore[arg-type]
-            )
-            .unique()
-            .all()
-        )
-
-        # Batch-fetch SCIM mappings to avoid N+1 queries
-        mapping_map = self._get_user_mappings_batch([u.id for u in users])
-        return [(u, mapping_map.get(u.id)) for u in users], total
-
-    def sync_user_external_id(
-        self,
-        user_id: UUID,
-        new_external_id: str | None,
-        scim_username: str | None = None,
-        fields: ScimMappingFields | None = None,
-    ) -> None:
-        """Sync the SCIM mapping for a user.
-
-        If a mapping already exists, its fields are updated (including
-        setting ``external_id`` to ``None`` when the IdP omits it).
-        If no mapping exists and ``new_external_id`` is provided, a new
-        mapping is created.  A mapping is never deleted here — SCIM-managed
-        users must retain their mapping to remain visible in ``GET /Users``.
-
-        When *fields* is provided, all mapping fields are written
-        unconditionally — including ``None`` values — so that a caller can
-        clear a previously-set field (e.g. removing a department).
-        """
-        mapping = self.get_user_mapping_by_user_id(user_id)
-        if mapping:
-            if mapping.external_id != new_external_id:
-                mapping.external_id = new_external_id
-            if scim_username is not None:
-                mapping.scim_username = scim_username
-            if fields is not None:
-                mapping.department = fields.department
-                mapping.manager = fields.manager
-                mapping.given_name = fields.given_name
-                mapping.family_name = fields.family_name
-                mapping.scim_emails_json = fields.scim_emails_json
-        elif new_external_id:
-            self.create_user_mapping(
-                external_id=new_external_id,
-                user_id=user_id,
-                scim_username=scim_username,
-                fields=fields,
-            )
-
-    def _get_user_mappings_batch(
-        self, user_ids: list[UUID]
-    ) -> dict[UUID, ScimUserMapping]:
-        """Batch-fetch SCIM user mappings keyed by user ID."""
-        if not user_ids:
-            return {}
-        mappings = self._session.scalars(
-            select(ScimUserMapping).where(ScimUserMapping.user_id.in_(user_ids))
-        ).all()
-        return {m.user_id: m for m in mappings}
-
-    def get_user_groups(self, user_id: UUID) -> list[tuple[int, str]]:
-        """Get groups a user belongs to as ``(group_id, group_name)`` pairs.
-
-        Excludes groups marked for deletion.
-        """
-        rels = self._session.scalars(
-            select(User__UserGroup).where(User__UserGroup.user_id == user_id)
-        ).all()
-
-        group_ids = [r.user_group_id for r in rels]
-        if not group_ids:
-            return []
-
-        groups = self._session.scalars(
-            select(UserGroup).where(
-                UserGroup.id.in_(group_ids),
-                UserGroup.is_up_for_deletion.is_(False),
-            )
-        ).all()
-        return [(g.id, g.name) for g in groups]
-
-    def get_users_groups_batch(
-        self, user_ids: list[UUID]
-    ) -> dict[UUID, list[tuple[int, str]]]:
-        """Batch-fetch group memberships for multiple users.
-
-        Returns a mapping of ``user_id → [(group_id, group_name), ...]``.
-        Avoids N+1 queries when building user list responses.
-        """
-        if not user_ids:
-            return {}
-
-        rels = self._session.scalars(
-            select(User__UserGroup).where(User__UserGroup.user_id.in_(user_ids))
-        ).all()
-
-        group_ids = list({r.user_group_id for r in rels})
-        if not group_ids:
-            return {}
-
-        groups = self._session.scalars(
-            select(UserGroup).where(
-                UserGroup.id.in_(group_ids),
-                UserGroup.is_up_for_deletion.is_(False),
-            )
-        ).all()
-        groups_by_id = {g.id: g.name for g in groups}
-
-        result: dict[UUID, list[tuple[int, str]]] = {}
-        for r in rels:
-            if r.user_id and r.user_group_id in groups_by_id:
-                result.setdefault(r.user_id, []).append(
-                    (r.user_group_id, groups_by_id[r.user_group_id])
-                )
-        return result
-
-    # ------------------------------------------------------------------
-    # Group mapping operations
-    # ------------------------------------------------------------------
-
-    def create_group_mapping(
-        self,
-        external_id: str,
-        user_group_id: int,
-    ) -> ScimGroupMapping:
-        """Create a mapping between a SCIM externalId and an Onyx user group."""
-        mapping = ScimGroupMapping(external_id=external_id, user_group_id=user_group_id)
-        self._session.add(mapping)
-        self._session.flush()
-        return mapping
-
-    def get_group_mapping_by_external_id(
-        self, external_id: str
-    ) -> ScimGroupMapping | None:
-        """Look up a group mapping by the IdP's external identifier."""
-        return self._session.scalar(
-            select(ScimGroupMapping).where(ScimGroupMapping.external_id == external_id)
-        )
-
-    def get_group_mapping_by_group_id(
-        self, user_group_id: int
-    ) -> ScimGroupMapping | None:
-        """Look up a group mapping by the Onyx user group ID."""
-        return self._session.scalar(
-            select(ScimGroupMapping).where(
-                ScimGroupMapping.user_group_id == user_group_id
-            )
-        )
-
-    def list_group_mappings(
-        self,
-        start_index: int = 1,
-        count: int = 100,
-    ) -> tuple[list[ScimGroupMapping], int]:
-        """List group mappings with SCIM-style pagination.
-
-        Args:
-            start_index: 1-based start index (SCIM convention).
-            count: Maximum number of results to return.
-
-        Returns:
-            A tuple of (mappings, total_count).
-        """
-        total = (
-            self._session.scalar(select(func.count()).select_from(ScimGroupMapping))
-            or 0
-        )
-
-        offset = max(start_index - 1, 0)
-        mappings = list(
-            self._session.scalars(
-                select(ScimGroupMapping)
-                .order_by(ScimGroupMapping.id)
-                .offset(offset)
-                .limit(count)
-            ).all()
-        )
-
-        return mappings, total
-
-    def delete_group_mapping(self, mapping_id: int) -> None:
-        """Delete a group mapping by ID. No-op if already deleted."""
-        mapping = self._session.get(ScimGroupMapping, mapping_id)
-        if not mapping:
-            logger.warning("SCIM group mapping %d not found during delete", mapping_id)
-            return
-        self._session.delete(mapping)
-
-    # ------------------------------------------------------------------
-    # Group query operations
-    # ------------------------------------------------------------------
-
-    def get_group(self, group_id: int) -> UserGroup | None:
-        """Fetch a group by ID, returning None if deleted or missing."""
-        group = self._session.get(UserGroup, group_id)
-        if group and group.is_up_for_deletion:
-            return None
-        return group
-
-    def get_group_by_name(self, name: str) -> UserGroup | None:
-        """Fetch a group by exact name."""
-        return self._session.scalar(select(UserGroup).where(UserGroup.name == name))
-
-    def add_group(self, group: UserGroup) -> None:
-        """Add a new group to the session and flush to assign an ID."""
-        self._session.add(group)
-        self._session.flush()
-
-    def update_group(
-        self,
-        group: UserGroup,
-        *,
-        name: str | None = None,
-    ) -> None:
-        """Update group attributes and set the modification timestamp."""
-        if name is not None:
-            group.name = name
-        group.time_last_modified_by_user = func.now()
-
-    def delete_group(self, group: UserGroup) -> None:
-        """Delete a group from the session."""
-        self._session.delete(group)
-
-    def list_groups(
-        self,
-        scim_filter: ScimFilter | None,
-        start_index: int = 1,
-        count: int = 100,
-    ) -> tuple[list[tuple[UserGroup, str | None]], int]:
-        """Query groups with optional SCIM filter and pagination.
-
-        Returns:
-            A tuple of (list of (group, external_id) pairs, total_count).
-
-        Raises:
-            ValueError: If the filter uses an unsupported attribute.
-        """
-        query = select(UserGroup).where(UserGroup.is_up_for_deletion.is_(False))
-
-        if scim_filter:
-            attr = scim_filter.attribute.lower()
-            if attr == "displayname":
-                # assignment: union return type widens but query is still Select[tuple[UserGroup]]
-                query = _apply_scim_string_op(query, UserGroup.name, scim_filter)  # type: ignore[assignment]
-            elif attr == "externalid":
-                mapping = self.get_group_mapping_by_external_id(scim_filter.value)
-                if not mapping:
-                    return [], 0
-                query = query.where(UserGroup.id == mapping.user_group_id)
-            else:
-                raise ValueError(
-                    f"Unsupported filter attribute: {scim_filter.attribute}"
-                )
-
-        total = (
-            self._session.scalar(select(func.count()).select_from(query.subquery()))
-            or 0
-        )
-
-        offset = max(start_index - 1, 0)
-        groups = list(
-            self._session.scalars(
-                query.order_by(UserGroup.id).offset(offset).limit(count)
-            ).all()
-        )
-
-        ext_id_map = self._get_group_external_ids([g.id for g in groups])
-        return [(g, ext_id_map.get(g.id)) for g in groups], total
-
-    def get_group_members(self, group_id: int) -> list[tuple[UUID, str | None]]:
-        """Get group members as (user_id, email) pairs."""
-        rels = self._session.scalars(
-            select(User__UserGroup).where(User__UserGroup.user_group_id == group_id)
-        ).all()
-
-        user_ids = [r.user_id for r in rels if r.user_id]
-        if not user_ids:
-            return []
-
-        users = (
-            self._session.scalars(
-                select(User).where(User.id.in_(user_ids))  # type: ignore[attr-defined]
-            )
-            .unique()
-            .all()
-        )
-        users_by_id = {u.id: u for u in users}
-
-        return [
-            (
-                r.user_id,
-                users_by_id[r.user_id].email if r.user_id in users_by_id else None,
-            )
-            for r in rels
-            if r.user_id
-        ]
-
-    def validate_member_ids(self, uuids: list[UUID]) -> list[UUID]:
-        """Return the subset of UUIDs that don't exist as users.
-
-        Returns an empty list if all IDs are valid.
-        """
-        if not uuids:
-            return []
-        existing_users = (
-            self._session.scalars(
-                select(User).where(User.id.in_(uuids))  # type: ignore[attr-defined]
-            )
-            .unique()
-            .all()
-        )
-        existing_ids = {u.id for u in existing_users}
-        return [uid for uid in uuids if uid not in existing_ids]
-
-    def upsert_group_members(self, group_id: int, user_ids: list[UUID]) -> None:
-        """Add user-group relationships, ignoring duplicates."""
-        if not user_ids:
-            return
-        self._session.execute(
-            pg_insert(User__UserGroup)
-            .values([{"user_id": uid, "user_group_id": group_id} for uid in user_ids])
-            .on_conflict_do_nothing(
-                index_elements=[
-                    User__UserGroup.user_group_id,
-                    User__UserGroup.user_id,
-                ]
-            )
-        )
-
-    def replace_group_members(self, group_id: int, user_ids: list[UUID]) -> None:
-        """Replace all members of a group."""
-        self._session.execute(
-            sa_delete(User__UserGroup).where(User__UserGroup.user_group_id == group_id)
-        )
-        self.upsert_group_members(group_id, user_ids)
-
-    def remove_group_members(self, group_id: int, user_ids: list[UUID]) -> None:
-        """Remove specific members from a group."""
-        if not user_ids:
-            return
-        self._session.execute(
-            sa_delete(User__UserGroup).where(
-                User__UserGroup.user_group_id == group_id,
-                User__UserGroup.user_id.in_(user_ids),
-            )
-        )
-
-    def delete_group_with_members(self, group: UserGroup) -> None:
-        """Remove all member relationships and delete the group."""
-        self._session.execute(
-            sa_delete(User__UserGroup).where(User__UserGroup.user_group_id == group.id)
-        )
-        self._session.delete(group)
-
-    def sync_group_external_id(
-        self, group_id: int, new_external_id: str | None
-    ) -> None:
-        """Create, update, or delete the external ID mapping for a group."""
-        mapping = self.get_group_mapping_by_group_id(group_id)
-        if new_external_id:
-            if mapping:
-                if mapping.external_id != new_external_id:
-                    mapping.external_id = new_external_id
-            else:
-                self.create_group_mapping(
-                    external_id=new_external_id, user_group_id=group_id
-                )
-        elif mapping:
-            self.delete_group_mapping(mapping.id)
-
-    def _get_group_external_ids(self, group_ids: list[int]) -> dict[int, str]:
-        """Batch-fetch external IDs for a list of group IDs."""
-        if not group_ids:
-            return {}
-        mappings = self._session.scalars(
-            select(ScimGroupMapping).where(
-                ScimGroupMapping.user_group_id.in_(group_ids)
-            )
-        ).all()
-        return {m.user_group_id: m.external_id for m in mappings}
-
-
-# ---------------------------------------------------------------------------
-# Module-level helpers (used by DAL methods above)
-# ---------------------------------------------------------------------------
-
-
-def _apply_scim_string_op(
-    query: Select[tuple[User]] | Select[tuple[UserGroup]],
-    column: SQLColumnExpression[str],
-    scim_filter: ScimFilter,
-) -> Select[tuple[User]] | Select[tuple[UserGroup]]:
-    """Apply a SCIM string filter operator using SQLAlchemy column operators.
-
-    Handles eq (case-insensitive exact), co (contains), and sw (starts with).
-    SQLAlchemy's operators handle LIKE-pattern escaping internally.
-    """
-    val = scim_filter.value
-    if scim_filter.operator == ScimFilterOperator.EQUAL:
-        return query.where(func.lower(column) == val.lower())
-    elif scim_filter.operator == ScimFilterOperator.CONTAINS:
-        return query.where(column.icontains(val, autoescape=True))
-    elif scim_filter.operator == ScimFilterOperator.STARTS_WITH:
-        return query.where(column.istartswith(val, autoescape=True))
-    else:
-        raise ValueError(f"Unsupported string filter operator: {scim_filter.operator}")

backend/ee/onyx/db/standard_answer.py

@@ -191,7 +191,8 @@ def create_initial_default_standard_answer_category(db_session: Session) -> None
     if default_category is not None:
         if default_category.name != default_category_name:
             raise ValueError(
-                "DB is not in a valid initial state. Default standard answer category does not have expected name."
+                "DB is not in a valid initial state. "
+                "Default standard answer category does not have expected name."
             )
         return
 

backend/ee/onyx/db/user_group.py

@@ -9,26 +9,20 @@ from sqlalchemy import Select
 from sqlalchemy import select
 from sqlalchemy import update
 from sqlalchemy.dialects.postgresql import insert
-from sqlalchemy.orm import selectinload
 from sqlalchemy.orm import Session
 
 from ee.onyx.server.user_group.models import SetCuratorRequest
 from ee.onyx.server.user_group.models import UserGroupCreate
 from ee.onyx.server.user_group.models import UserGroupUpdate
-from onyx.configs.app_configs import DISABLE_VECTOR_DB
 from onyx.db.connector_credential_pair import get_connector_credential_pair_from_id
 from onyx.db.enums import AccessType
 from onyx.db.enums import ConnectorCredentialPairStatus
 from onyx.db.models import ConnectorCredentialPair
-from onyx.db.models import Credential
 from onyx.db.models import Credential__UserGroup
 from onyx.db.models import Document
 from onyx.db.models import DocumentByConnectorCredentialPair
-from onyx.db.models import DocumentSet
 from onyx.db.models import DocumentSet__UserGroup
-from onyx.db.models import FederatedConnector__DocumentSet
 from onyx.db.models import LLMProvider__UserGroup
-from onyx.db.models import Persona
 from onyx.db.models import Persona__UserGroup
 from onyx.db.models import TokenRateLimit__UserGroup
 from onyx.db.models import User
@@ -201,60 +195,8 @@ def fetch_user_group(db_session: Session, user_group_id: int) -> UserGroup | Non
     return db_session.scalar(stmt)
 
 
-def _add_user_group_snapshot_eager_loads(
-    stmt: Select,
-) -> Select:
-    """Add eager loading options needed by UserGroup.from_model snapshot creation."""
-    return stmt.options(
-        selectinload(UserGroup.users),
-        selectinload(UserGroup.user_group_relationships),
-        selectinload(UserGroup.cc_pair_relationships)
-        .selectinload(UserGroup__ConnectorCredentialPair.cc_pair)
-        .options(
-            selectinload(ConnectorCredentialPair.connector),
-            selectinload(ConnectorCredentialPair.credential).selectinload(
-                Credential.user
-            ),
-        ),
-        selectinload(UserGroup.document_sets).options(
-            selectinload(DocumentSet.connector_credential_pairs).selectinload(
-                ConnectorCredentialPair.connector
-            ),
-            selectinload(DocumentSet.users),
-            selectinload(DocumentSet.groups),
-            selectinload(DocumentSet.federated_connectors).selectinload(
-                FederatedConnector__DocumentSet.federated_connector
-            ),
-        ),
-        selectinload(UserGroup.personas).options(
-            selectinload(Persona.tools),
-            selectinload(Persona.hierarchy_nodes),
-            selectinload(Persona.attached_documents).selectinload(
-                Document.parent_hierarchy_node
-            ),
-            selectinload(Persona.labels),
-            selectinload(Persona.document_sets).options(
-                selectinload(DocumentSet.connector_credential_pairs).selectinload(
-                    ConnectorCredentialPair.connector
-                ),
-                selectinload(DocumentSet.users),
-                selectinload(DocumentSet.groups),
-                selectinload(DocumentSet.federated_connectors).selectinload(
-                    FederatedConnector__DocumentSet.federated_connector
-                ),
-            ),
-            selectinload(Persona.user),
-            selectinload(Persona.user_files),
-            selectinload(Persona.users),
-            selectinload(Persona.groups),
-        ),
-    )
-
-
 def fetch_user_groups(
-    db_session: Session,
-    only_up_to_date: bool = True,
-    eager_load_for_snapshot: bool = False,
+    db_session: Session, only_up_to_date: bool = True
 ) -> Sequence[UserGroup]:
     """
     Fetches user groups from the database.
@@ -267,8 +209,6 @@ def fetch_user_groups(
         db_session (Session): The SQLAlchemy session used to query the database.
         only_up_to_date (bool, optional): Flag to determine whether to filter the results
             to include only up to date user groups. Defaults to `True`.
-        eager_load_for_snapshot: If True, adds eager loading for all relationships
-            needed by UserGroup.from_model snapshot creation.
 
     Returns:
         Sequence[UserGroup]: A sequence of `UserGroup` objects matching the query criteria.
@@ -276,16 +216,11 @@ def fetch_user_groups(
     stmt = select(UserGroup)
     if only_up_to_date:
         stmt = stmt.where(UserGroup.is_up_to_date == True)  # noqa: E712
-    if eager_load_for_snapshot:
-        stmt = _add_user_group_snapshot_eager_loads(stmt)
-    return db_session.scalars(stmt).unique().all()
+    return db_session.scalars(stmt).all()
 
 
 def fetch_user_groups_for_user(
-    db_session: Session,
-    user_id: UUID,
-    only_curator_groups: bool = False,
-    eager_load_for_snapshot: bool = False,
+    db_session: Session, user_id: UUID, only_curator_groups: bool = False
 ) -> Sequence[UserGroup]:
     stmt = (
         select(UserGroup)
@@ -295,9 +230,7 @@ def fetch_user_groups_for_user(
     )
     if only_curator_groups:
         stmt = stmt.where(User__UserGroup.is_curator == True)  # noqa: E712
-    if eager_load_for_snapshot:
-        stmt = _add_user_group_snapshot_eager_loads(stmt)
-    return db_session.scalars(stmt).unique().all()
+    return db_session.scalars(stmt).all()
 
 
 def construct_document_id_select_by_usergroup(
@@ -424,7 +357,8 @@ def fetch_user_groups_for_documents(
 def _check_user_group_is_modifiable(user_group: UserGroup) -> None:
     if not user_group.is_up_to_date:
         raise ValueError(
-            "Specified user group is currently syncing. Wait until the current sync has finished before editing."
+            "Specified user group is currently syncing. Wait until the current "
+            "sync has finished before editing."
         )
 
 
@@ -471,9 +405,7 @@ def _add_user_group__cc_pair_relationships__no_commit(
 
 def insert_user_group(db_session: Session, user_group: UserGroupCreate) -> UserGroup:
     db_user_group = UserGroup(
-        name=user_group.name,
-        time_last_modified_by_user=func.now(),
-        is_up_to_date=DISABLE_VECTOR_DB,
+        name=user_group.name, time_last_modified_by_user=func.now()
     )
     db_session.add(db_user_group)
     db_session.flush()  # give the group an ID
@@ -776,7 +708,8 @@ def update_user_group(
             cc_pair_ids=user_group_update.cc_pair_ids,
         )
 
-    if cc_pairs_updated and not DISABLE_VECTOR_DB:
+    # only needs to sync with Vespa if the cc_pairs have been updated
+    if cc_pairs_updated:
         db_user_group.is_up_to_date = False
 
     removed_users = db_session.scalars(

backend/ee/onyx/external_permissions/github/utils.py

@@ -56,7 +56,8 @@ def _run_with_retry(
         if retry_count < MAX_RETRY_COUNT:
             sleep_after_rate_limit_exception(github_client)
             logger.warning(
-                f"Rate limit exceeded while {description}. Retrying... (attempt {retry_count + 1}/{MAX_RETRY_COUNT})"
+                f"Rate limit exceeded while {description}. Retrying... "
+                f"(attempt {retry_count + 1}/{MAX_RETRY_COUNT})"
             )
             return _run_with_retry(
                 operation, description, github_client, retry_count + 1
@@ -90,9 +91,7 @@ class TeamInfo(BaseModel):
 
 
 def _fetch_organization_members(
-    github_client: Github,
-    org_name: str,
-    retry_count: int = 0,  # noqa: ARG001
+    github_client: Github, org_name: str, retry_count: int = 0  # noqa: ARG001
 ) -> List[UserInfo]:
     """Fetch all organization members including owners and regular members."""
     org_members: List[UserInfo] = []
@@ -125,9 +124,7 @@ def _fetch_organization_members(
 
 
 def _fetch_repository_teams_detailed(
-    repo: Repository,
-    github_client: Github,
-    retry_count: int = 0,  # noqa: ARG001
+    repo: Repository, github_client: Github, retry_count: int = 0  # noqa: ARG001
 ) -> List[TeamInfo]:
     """Fetch teams with access to the repository and their members."""
     teams_data: List[TeamInfo] = []
@@ -170,9 +167,7 @@ def _fetch_repository_teams_detailed(
 
 
 def fetch_repository_team_slugs(
-    repo: Repository,
-    github_client: Github,
-    retry_count: int = 0,  # noqa: ARG001
+    repo: Repository, github_client: Github, retry_count: int = 0  # noqa: ARG001
 ) -> List[str]:
     """Fetch team slugs with access to the repository."""
     logger.info(f"Fetching team slugs for repository {repo.full_name}")

backend/ee/onyx/external_permissions/google_drive/doc_sync.py

@@ -68,7 +68,6 @@ def get_external_access_for_raw_gdrive_file(
     company_domain: str,
     retriever_drive_service: GoogleDriveService | None,
     admin_drive_service: GoogleDriveService,
-    fallback_user_email: str,
     add_prefix: bool = False,
 ) -> ExternalAccess:
     """
@@ -80,11 +79,6 @@ def get_external_access_for_raw_gdrive_file(
                 set add_prefix to True so group IDs are prefixed with the source type.
                 When invoked from doc_sync (permission sync), use the default (False)
                 since upsert_document_external_perms handles prefixing.
-    fallback_user_email: When we cannot retrieve any permission info for a file
-                (e.g. externally-owned files where the API returns no permissions
-                and permissions.list returns 403), fall back to granting access
-                to this user. This is typically the impersonated org user whose
-                drive contained the file.
     """
     doc_id = file.get("id")
     if not doc_id:
@@ -115,33 +109,14 @@ def get_external_access_for_raw_gdrive_file(
         )
         if len(permissions_list) != len(permission_ids) and retriever_drive_service:
             logger.warning(
-                f"Failed to get all permissions for file {doc_id} with retriever service, trying admin service"
+                f"Failed to get all permissions for file {doc_id} with retriever service, "
+                "trying admin service"
             )
             backup_permissions_list = _get_permissions(admin_drive_service)
             permissions_list = _merge_permissions_lists(
                 [permissions_list, backup_permissions_list]
             )
 
-    # For externally-owned files, the Drive API may return no permissions
-    # and permissions.list may return 403. In this case, fall back to
-    # granting access to the user who found the file in their drive.
-    # Note, even if other users also have access to this file,
-    # they will not be granted access in Onyx.
-    # We check permissions_list (the final result after all fetch attempts)
-    # rather than the raw fields, because permission_ids may be present
-    # but the actual fetch can still return empty due to a 403.
-    if not permissions_list:
-        logger.info(
-            f"No permission info available for file {doc_id} "
-            f"(likely owned by a user outside of your organization). "
-            f"Falling back to granting access to retriever user: {fallback_user_email}"
-        )
-        return ExternalAccess(
-            external_user_emails={fallback_user_email},
-            external_user_group_ids=set(),
-            is_public=False,
-        )
-
     folder_ids_to_inherit_permissions_from: set[str] = set()
     user_emails: set[str] = set()
     group_emails: set[str] = set()
@@ -165,7 +140,9 @@ def get_external_access_for_raw_gdrive_file(
                 user_emails.add(permission.email_address)
             else:
                 logger.error(
-                    f"Permission is type `user` but no email address is provided for document {doc_id}\n {permission}"
+                    "Permission is type `user` but no email address is "
+                    f"provided for document {doc_id}"
+                    f"\n {permission}"
                 )
         elif permission.type == PermissionType.GROUP:
             # groups are represented as email addresses within Drive
@@ -173,14 +150,17 @@ def get_external_access_for_raw_gdrive_file(
                 group_emails.add(permission.email_address)
             else:
                 logger.error(
-                    f"Permission is type `group` but no email address is provided for document {doc_id}\n {permission}"
+                    "Permission is type `group` but no email address is "
+                    f"provided for document {doc_id}"
+                    f"\n {permission}"
                 )
         elif permission.type == PermissionType.DOMAIN and company_domain:
             if permission.domain == company_domain:
                 public = True
             else:
                 logger.warning(
-                    f"Permission is type domain but does not match company domain:\n {permission}"
+                    "Permission is type domain but does not match company domain:"
+                    f"\n {permission}"
                 )
         elif permission.type == PermissionType.ANYONE:
             public = True

backend/ee/onyx/external_permissions/google_drive/folder_retrieval.py

@@ -18,7 +18,10 @@ logger = setup_logger()
 # Only include fields we need - folder ID and permissions
 # IMPORTANT: must fetch permissionIds, since sometimes the drive API
 # seems to miss permissions when requesting them directly
-FOLDER_PERMISSION_FIELDS = "nextPageToken, files(id, name, permissionIds, permissions(id, emailAddress, type, domain, permissionDetails))"
+FOLDER_PERMISSION_FIELDS = (
+    "nextPageToken, files(id, name, permissionIds, "
+    "permissions(id, emailAddress, type, domain, permissionDetails))"
+)
 
 
 def get_folder_permissions_by_ids(

backend/ee/onyx/external_permissions/google_drive/group_sync.py

@@ -142,7 +142,8 @@ def _drive_folder_to_onyx_group(
         elif permission.type == PermissionType.GROUP:
             if permission.email_address not in group_email_to_member_emails_map:
                 logger.warning(
-                    f"Group email {permission.email_address} for folder {folder.id} not found in group_email_to_member_emails_map"
+                    f"Group email {permission.email_address} for folder {folder.id} "
+                    "not found in group_email_to_member_emails_map"
                 )
                 continue
             folder_member_emails.update(
@@ -237,7 +238,8 @@ def _drive_member_map_to_onyx_groups(
         for group_email in group_emails:
             if group_email not in group_email_to_member_emails_map:
                 logger.warning(
-                    f"Group email {group_email} for drive {drive_id} not found in group_email_to_member_emails_map"
+                    f"Group email {group_email} for drive {drive_id} not found in "
+                    "group_email_to_member_emails_map"
                 )
                 continue
             drive_member_emails.update(group_email_to_member_emails_map[group_email])
@@ -324,7 +326,8 @@ def _build_onyx_groups(
         for group_email in group_emails:
             if group_email not in group_email_to_member_emails_map:
                 logger.warning(
-                    f"Group email {group_email} for drive {drive_id} not found in group_email_to_member_emails_map"
+                    f"Group email {group_email} for drive {drive_id} not found in "
+                    "group_email_to_member_emails_map"
                 )
                 continue
             drive_member_emails.update(group_email_to_member_emails_map[group_email])

backend/ee/onyx/external_permissions/google_drive/permission_retrieval.py

@@ -55,7 +55,8 @@ def get_permissions_by_ids(
     if len(filtered_permissions) < len(permission_ids):
         missing_ids = permission_id_set - {p.id for p in filtered_permissions if p.id}
         logger.warning(
-            f"Could not find all requested permission IDs for document {doc_id}. Missing IDs: {missing_ids}"
+            f"Could not find all requested permission IDs for document {doc_id}. "
+            f"Missing IDs: {missing_ids}"
         )
 
     return filtered_permissions

backend/ee/onyx/external_permissions/jira/group_sync.py

@@ -1,8 +1,6 @@
 from collections.abc import Generator
-from typing import Any
 
 from jira import JIRA
-from jira.exceptions import JIRAError
 
 from ee.onyx.db.external_perm import ExternalUserGroup
 from onyx.connectors.jira.utils import build_jira_client
@@ -11,101 +9,107 @@ from onyx.utils.logger import setup_logger
 
 logger = setup_logger()
 
-_ATLASSIAN_ACCOUNT_TYPE = "atlassian"
-_GROUP_MEMBER_PAGE_SIZE = 50
 
-# The GET /group/member endpoint was introduced in Jira 6.0.
-# Jira versions older than 6.0 do not have group management REST APIs at all.
-_MIN_JIRA_VERSION_FOR_GROUP_MEMBER = "6.0"
-
-
-def _fetch_group_member_page(
+def _get_jira_group_members_email(
     jira_client: JIRA,
     group_name: str,
-    start_at: int,
-) -> dict[str, Any]:
-    """Fetch a single page from the non-deprecated GET /group/member endpoint.
+) -> list[str]:
+    """Get all member emails for a Jira group.
 
-    The old GET /group endpoint (used by jira_client.group_members()) is deprecated
-    and decommissioned in Jira Server 10.3+. This uses the replacement endpoint
-    directly via the library's internal _get_json helper, following the same pattern
-    as enhanced_search_ids / bulk_fetch_issues in connector.py.
-
-    There is an open PR to the library to switch to this endpoint since last year:
-    https://github.com/pycontribs/jira/pull/2356
-    so once it is merged and released, we can switch to using the library function.
+    Filters out app accounts (bots, integrations) and only returns real user emails.
     """
+    emails: list[str] = []
+
     try:
-        return jira_client._get_json(
-            "group/member",
-            params={
-                "groupname": group_name,
-                "includeInactiveUsers": "false",
-                "startAt": start_at,
-                "maxResults": _GROUP_MEMBER_PAGE_SIZE,
-            },
-        )
-    except JIRAError as e:
-        if e.status_code == 404:
-            raise RuntimeError(
-                f"GET /group/member returned 404 for group '{group_name}'. "
-                f"This endpoint requires Jira {_MIN_JIRA_VERSION_FOR_GROUP_MEMBER}+. "
-                f"If you are running a self-hosted Jira instance, please upgrade "
-                f"to at least Jira {_MIN_JIRA_VERSION_FOR_GROUP_MEMBER}."
-            ) from e
-        raise
+        # group_members returns an OrderedDict of account_id -> member_info
+        members = jira_client.group_members(group=group_name)
 
+        if not members:
+            logger.warning(f"No members found for group {group_name}")
+            return emails
 
-def _get_group_member_emails(
-    jira_client: JIRA,
-    group_name: str,
-) -> set[str]:
-    """Get all member emails for a single Jira group.
+        for account_id, member_info in members.items():
+            # member_info is a dict with keys like 'fullname', 'email', 'active'
+            email = member_info.get("email")
 
-    Uses the non-deprecated GET /group/member endpoint which returns full user
-    objects including accountType, so we can filter out app/customer accounts
-    without making separate user() calls.
-    """
-    emails: set[str] = set()
-    start_at = 0
-
-    while True:
-        try:
-            page = _fetch_group_member_page(jira_client, group_name, start_at)
-        except Exception as e:
-            logger.error(f"Error fetching members for group {group_name}: {e}")
-            raise
-
-        members: list[dict[str, Any]] = page.get("values", [])
-        for member in members:
-            account_type = member.get("accountType")
-            # On Jira DC < 9.0, accountType is absent; include those users.
-            # On Cloud / DC 9.0+, filter to real user accounts only.
-            if account_type is not None and account_type != _ATLASSIAN_ACCOUNT_TYPE:
-                continue
-
-            email = member.get("emailAddress")
-            if email:
-                emails.add(email)
+            # Skip "hidden" emails - these are typically app accounts
+            if email and email != "hidden":
+                emails.append(email)
             else:
-                logger.warning(
-                    f"Atlassian user {member.get('accountId', 'unknown')} in group {group_name} has no visible email address"
-                )
+                # For cloud, we might need to fetch user details separately
+                try:
+                    user = jira_client.user(id=account_id)
 
-        if page.get("isLast", True) or not members:
-            break
-        start_at += len(members)
+                    # Skip app accounts (bots, integrations, etc.)
+                    if hasattr(user, "accountType") and user.accountType == "app":
+                        logger.info(
+                            f"Skipping app account {account_id} for group {group_name}"
+                        )
+                        continue
+
+                    if hasattr(user, "emailAddress") and user.emailAddress:
+                        emails.append(user.emailAddress)
+                    else:
+                        logger.warning(f"User {account_id} has no email address")
+                except Exception as e:
+                    logger.warning(
+                        f"Could not fetch email for user {account_id} in group {group_name}: {e}"
+                    )
+
+    except Exception as e:
+        logger.error(f"Error fetching members for group {group_name}: {e}")
 
     return emails
 
 
+def _build_group_member_email_map(
+    jira_client: JIRA,
+) -> dict[str, set[str]]:
+    """Build a map of group names to member emails."""
+    group_member_emails: dict[str, set[str]] = {}
+
+    try:
+        # Get all groups from Jira - returns a list of group name strings
+        group_names = jira_client.groups()
+
+        if not group_names:
+            logger.warning("No groups found in Jira")
+            return group_member_emails
+
+        logger.info(f"Found {len(group_names)} groups in Jira")
+
+        for group_name in group_names:
+            if not group_name:
+                continue
+
+            member_emails = _get_jira_group_members_email(
+                jira_client=jira_client,
+                group_name=group_name,
+            )
+
+            if member_emails:
+                group_member_emails[group_name] = set(member_emails)
+                logger.debug(
+                    f"Found {len(member_emails)} members for group {group_name}"
+                )
+            else:
+                logger.debug(f"No members found for group {group_name}")
+
+    except Exception as e:
+        logger.error(f"Error building group member email map: {e}")
+
+    return group_member_emails
+
+
 def jira_group_sync(
     tenant_id: str,  # noqa: ARG001
     cc_pair: ConnectorCredentialPair,
 ) -> Generator[ExternalUserGroup, None, None]:
-    """Sync Jira groups and their members, yielding one group at a time.
+    """
+    Sync Jira groups and their members.
 
-    Streams group-by-group rather than accumulating all groups in memory.
+    This function fetches all groups from Jira and yields ExternalUserGroup
+    objects containing the group ID and member emails.
     """
     jira_base_url = cc_pair.connector.connector_specific_config.get("jira_base_url", "")
     scoped_token = cc_pair.connector.connector_specific_config.get(
@@ -126,26 +130,12 @@ def jira_group_sync(
         scoped_token=scoped_token,
     )
 
-    group_names = jira_client.groups()
-    if not group_names:
-        raise ValueError(f"No groups found for cc_pair_id={cc_pair.id}")
+    group_member_email_map = _build_group_member_email_map(jira_client=jira_client)
+    if not group_member_email_map:
+        raise ValueError(f"No groups with members found for cc_pair_id={cc_pair.id}")
 
-    logger.info(f"Found {len(group_names)} groups in Jira")
-
-    for group_name in group_names:
-        if not group_name:
-            continue
-
-        member_emails = _get_group_member_emails(
-            jira_client=jira_client,
-            group_name=group_name,
-        )
-        if not member_emails:
-            logger.debug(f"No members found for group {group_name}")
-            continue
-
-        logger.debug(f"Found {len(member_emails)} members for group {group_name}")
+    for group_id, group_member_emails in group_member_email_map.items():
         yield ExternalUserGroup(
-            id=group_name,
-            user_emails=list(member_emails),
+            id=group_id,
+            user_emails=list(group_member_emails),
         )

backend/ee/onyx/external_permissions/post_query_censoring.py

@@ -69,7 +69,8 @@ def _post_query_chunk_censoring(
             censored_chunks = censor_chunks_for_source(chunks_for_source, user.email)
         except Exception as e:
             logger.exception(
-                f"Failed to censor chunks for source {source} so throwing out all chunks for this source and continuing: {e}"
+                f"Failed to censor chunks for source {source} so throwing out all"
+                f" chunks for this source and continuing: {e}"
             )
             continue