build org

.github/workflows/pr-python-connector-tests.yml

@@ -24,8 +24,6 @@ env:
   GOOGLE_DRIVE_OAUTH_CREDENTIALS_JSON_STR: ${{ secrets.GOOGLE_DRIVE_OAUTH_CREDENTIALS_JSON_STR }}
   GOOGLE_GMAIL_SERVICE_ACCOUNT_JSON_STR: ${{ secrets.GOOGLE_GMAIL_SERVICE_ACCOUNT_JSON_STR }}
   GOOGLE_GMAIL_OAUTH_CREDENTIALS_JSON_STR: ${{ secrets.GOOGLE_GMAIL_OAUTH_CREDENTIALS_JSON_STR }}
-  # Slab
-  SLAB_BOT_TOKEN: ${{ secrets.SLAB_BOT_TOKEN }}
 
 jobs:
   connectors-check:

README.md

@@ -1,48 +1,48 @@
-<!-- DANSWER_METADATA={"link": "https://github.com/onyx-dot-app/onyx/blob/main/README.md"} -->
+<!-- DANSWER_METADATA={"link": "https://github.com/danswer-ai/danswer/blob/main/README.md"} -->
 <a name="readme-top"></a>
 
 <h2 align="center">
-<a href="https://www.onyx.app/"> <img width="50%" src="https://github.com/onyx-dot-app/onyx/blob/logo/LogoOnyx.png?raw=true)" /></a>
+<a href="https://www.danswer.ai/"> <img width="50%" src="https://github.com/danswer-owners/danswer/blob/1fabd9372d66cd54238847197c33f091a724803b/DanswerWithName.png?raw=true)" /></a>
 </h2>
 
 <p align="center">
-<p align="center">Open Source Gen-AI + Enterprise Search.</p>
+<p align="center">Open Source Gen-AI Chat + Unified Search.</p>
 
 <p align="center">
-<a href="https://docs.onyx.app/" target="_blank">
+<a href="https://docs.danswer.dev/" target="_blank">
     <img src="https://img.shields.io/badge/docs-view-blue" alt="Documentation">
 </a>
-<a href="https://join.slack.com/t/onyx-dot-app/shared_invite/zt-2sslpdbyq-iIbTaNIVPBw_i_4vrujLYQ" target="_blank">
+<a href="https://join.slack.com/t/danswer/shared_invite/zt-2twesxdr6-5iQitKZQpgq~hYIZ~dv3KA" target="_blank">
     <img src="https://img.shields.io/badge/slack-join-blue.svg?logo=slack" alt="Slack">
 </a>
 <a href="https://discord.gg/TDJ59cGV2X" target="_blank">
     <img src="https://img.shields.io/badge/discord-join-blue.svg?logo=discord&logoColor=white" alt="Discord">
 </a>
-<a href="https://github.com/onyx-dot-app/onyx/blob/main/README.md" target="_blank">
+<a href="https://github.com/danswer-ai/danswer/blob/main/README.md" target="_blank">
     <img src="https://img.shields.io/static/v1?label=license&message=MIT&color=blue" alt="License">
 </a>
 </p>
 
-<strong>[Onyx](https://www.onyx.app/)</strong> (Formerly Danswer) is the AI Assistant connected to your company's docs, apps, and people. 
-Onyx provides a Chat interface and plugs into any LLM of your choice. Onyx can be deployed anywhere and for any 
+<strong>[Danswer](https://www.danswer.ai/)</strong> is the AI Assistant connected to your company's docs, apps, and people. 
+Danswer provides a Chat interface and plugs into any LLM of your choice. Danswer can be deployed anywhere and for any 
 scale - on a laptop, on-premise, or to cloud. Since you own the deployment, your user data and chats are fully in your 
-own control. Onyx is dual Licensed with most of it under MIT license and designed to be modular and easily extensible. The system also comes fully ready 
+own control. Danswer is MIT licensed and designed to be modular and easily extensible. The system also comes fully ready 
 for production usage with user authentication, role management (admin/basic users), chat persistence, and a UI for 
-configuring AI Assistants.
+configuring Personas (AI Assistants) and their Prompts.
 
-Onyx also serves as a Enterprise Search across all common workplace tools such as Slack, Google Drive, Confluence, etc.
-By combining LLMs and team specific knowledge, Onyx becomes a subject matter expert for the team. Imagine ChatGPT if
+Danswer also serves as a Unified Search across all common workplace tools such as Slack, Google Drive, Confluence, etc.
+By combining LLMs and team specific knowledge, Danswer becomes a subject matter expert for the team. Imagine ChatGPT if
 it had access to your team's unique knowledge! It enables questions such as "A customer wants feature X, is this already
 supported?" or "Where's the pull request for feature Y?"
 
 <h3>Usage</h3>
 
-Onyx Web App:
+Danswer Web App:
 
 https://github.com/danswer-ai/danswer/assets/32520769/563be14c-9304-47b5-bf0a-9049c2b6f410
 
 
-Or, plug Onyx into your existing Slack workflows (more integrations to come 😁):
+Or, plug Danswer into your existing Slack workflows (more integrations to come 😁):
 
 https://github.com/danswer-ai/danswer/assets/25087905/3e19739b-d178-4371-9a38-011430bdec1b
 
@@ -52,16 +52,16 @@ For more details on the Admin UI to manage connectors and users, check out our
 
 ## Deployment
 
-Onyx can easily be run locally (even on a laptop) or deployed on a virtual machine with a single
-`docker compose` command. Checkout our [docs](https://docs.onyx.app/quickstart) to learn more.
+Danswer can easily be run locally (even on a laptop) or deployed on a virtual machine with a single
+`docker compose` command. Checkout our [docs](https://docs.danswer.dev/quickstart) to learn more.
 
-We also have built-in support for deployment on Kubernetes. Files for that can be found [here](https://github.com/onyx-dot-app/onyx/tree/main/deployment/kubernetes).
+We also have built-in support for deployment on Kubernetes. Files for that can be found [here](https://github.com/danswer-ai/danswer/tree/main/deployment/kubernetes).
 
 
 ## 💃 Main Features 
 * Chat UI with the ability to select documents to chat with.
 * Create custom AI Assistants with different prompts and backing knowledge sets.
-* Connect Onyx with LLM of your choice (self-host for a fully airgapped solution).
+* Connect Danswer with LLM of your choice (self-host for a fully airgapped solution).
 * Document Search + AI Answers for natural language queries.
 * Connectors to all common workplace tools like Google Drive, Confluence, Slack, etc.
 * Slack integration to get answers and search results directly in Slack.
@@ -75,12 +75,12 @@ We also have built-in support for deployment on Kubernetes. Files for that can b
 * Organizational understanding and ability to locate and suggest experts from your team.
 
 
-## Other Notable Benefits of Onyx
+## Other Notable Benefits of Danswer
 * User Authentication with document level access management.
 * Best in class Hybrid Search across all sources (BM-25 + prefix aware embedding models).
 * Admin Dashboard to configure connectors, document-sets, access, etc.
 * Custom deep learning models + learn from user feedback.
-* Easy deployment and ability to host Onyx anywhere of your choosing.
+* Easy deployment and ability to host Danswer anywhere of your choosing.
 
 
 ## 🔌 Connectors
@@ -108,10 +108,10 @@ Efficiently pulls the latest changes from:
 
 ## 📚 Editions
 
-There are two editions of Onyx:
+There are two editions of Danswer:
 
-  * Onyx Community Edition (CE) is available freely under the MIT Expat license. This version has ALL the core features discussed above. This is the version of Onyx you will get if you follow the Deployment guide above.
-  * Onyx Enterprise Edition (EE) includes extra features that are primarily useful for larger organizations. Specifically, this includes:
+  * Danswer Community Edition (CE) is available freely under the MIT Expat license. This version has ALL the core features discussed above. This is the version of Danswer you will get if you follow the Deployment guide above.
+  * Danswer Enterprise Edition (EE) includes extra features that are primarily useful for larger organizations. Specifically, this includes:
     * Single Sign-On (SSO), with support for both SAML and OIDC
     * Role-based access control
     * Document permission inheritance from connected sources
@@ -119,24 +119,24 @@ There are two editions of Onyx:
     * Whitelabeling
     * API key authentication
     * Encryption of secrets
-    * Any many more! Checkout [our website](https://www.onyx.app/) for the latest.
+    * Any many more! Checkout [our website](https://www.danswer.ai/) for the latest.
 
-To try the Onyx Enterprise Edition: 
+To try the Danswer Enterprise Edition: 
 
-  1. Checkout our [Cloud product](https://cloud.onyx.app/signup).
-  2. For self-hosting, contact us at [founders@onyx.app](mailto:founders@onyx.app) or book a call with us on our [Cal](https://cal.com/team/danswer/founders).
+  1. Checkout our [Cloud product](https://app.danswer.ai/signup).
+  2. For self-hosting, contact us at [founders@danswer.ai](mailto:founders@danswer.ai) or book a call with us on our [Cal](https://cal.com/team/danswer/founders).
 
 ## 💡 Contributing
 Looking to contribute? Please check out the [Contribution Guide](CONTRIBUTING.md) for more details.
 
 ## ⭐Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=onyx-dot-app/onyx&type=Date)](https://star-history.com/#onyx-dot-app/onyx&Date)
+[![Star History Chart](https://api.star-history.com/svg?repos=danswer-ai/danswer&type=Date)](https://star-history.com/#danswer-ai/danswer&Date)
 
 ## ✨Contributors
 
-<a href="https://github.com/onyx-dot-app/onyx/graphs/contributors">
-  <img alt="contributors" src="https://contrib.rocks/image?repo=onyx-dot-app/onyx"/>
+<a href="https://github.com/danswer-ai/danswer/graphs/contributors">
+  <img alt="contributors" src="https://contrib.rocks/image?repo=danswer-ai/danswer"/>
 </a>
 
 <p align="right" style="font-size: 14px; color: #555; margin-top: 20px;">

backend/Dockerfile

@@ -73,7 +73,6 @@ RUN apt-get update && \
     rm -rf /var/lib/apt/lists/* && \
     rm -f /usr/local/lib/python3.11/site-packages/tornado/test/test.key
 
-
 # Pre-downloading models for setups with limited egress
 RUN python -c "from tokenizers import Tokenizer; \
 Tokenizer.from_pretrained('nomic-ai/nomic-embed-text-v1')"

backend/alembic/env.py

@@ -1,5 +1,5 @@
 from sqlalchemy.engine.base import Connection
-from typing import Literal
+from typing import Any
 import asyncio
 from logging.config import fileConfig
 import logging
@@ -8,7 +8,6 @@ from alembic import context
 from sqlalchemy import pool
 from sqlalchemy.ext.asyncio import create_async_engine
 from sqlalchemy.sql import text
-from sqlalchemy.sql.schema import SchemaItem
 
 from shared_configs.configs import MULTI_TENANT
 from danswer.db.engine import build_connection_string
@@ -36,18 +35,7 @@ logger = logging.getLogger(__name__)
 
 
 def include_object(
-    object: SchemaItem,
-    name: str | None,
-    type_: Literal[
-        "schema",
-        "table",
-        "column",
-        "index",
-        "unique_constraint",
-        "foreign_key_constraint",
-    ],
-    reflected: bool,
-    compare_to: SchemaItem | None,
+    object: Any, name: str, type_: str, reflected: bool, compare_to: Any
 ) -> bool:
     """
     Determines whether a database object should be included in migrations.

backend/alembic/versions/9f696734098f_combine_search_and_chat.py

@@ -1,36 +0,0 @@
-"""Combine Search and Chat
-
-Revision ID: 9f696734098f
-Revises: a8c2065484e6
-Create Date: 2024-11-27 15:32:19.694972
-
-"""
-from alembic import op
-import sqlalchemy as sa
-
-# revision identifiers, used by Alembic.
-revision = "9f696734098f"
-down_revision = "a8c2065484e6"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.alter_column("chat_session", "description", nullable=True)
-    op.drop_column("chat_session", "one_shot")
-    op.drop_column("slack_channel_config", "response_type")
-
-
-def downgrade() -> None:
-    op.execute("UPDATE chat_session SET description = '' WHERE description IS NULL")
-    op.alter_column("chat_session", "description", nullable=False)
-    op.add_column(
-        "chat_session",
-        sa.Column("one_shot", sa.Boolean(), nullable=False, server_default=sa.false()),
-    )
-    op.add_column(
-        "slack_channel_config",
-        sa.Column(
-            "response_type", sa.String(), nullable=False, server_default="citations"
-        ),
-    )

backend/alembic/versions/bf7a81109301_delete_input_prompts.py

@@ -1,57 +0,0 @@
-"""delete_input_prompts
-
-Revision ID: bf7a81109301
-Revises: f7a894b06d02
-Create Date: 2024-12-09 12:00:49.884228
-
-"""
-from alembic import op
-import sqlalchemy as sa
-import fastapi_users_db_sqlalchemy
-
-
-# revision identifiers, used by Alembic.
-revision = "bf7a81109301"
-down_revision = "f7a894b06d02"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    op.drop_table("inputprompt__user")
-    op.drop_table("inputprompt")
-
-
-def downgrade() -> None:
-    op.create_table(
-        "inputprompt",
-        sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
-        sa.Column("prompt", sa.String(), nullable=False),
-        sa.Column("content", sa.String(), nullable=False),
-        sa.Column("active", sa.Boolean(), nullable=False),
-        sa.Column("is_public", sa.Boolean(), nullable=False),
-        sa.Column(
-            "user_id",
-            fastapi_users_db_sqlalchemy.generics.GUID(),
-            nullable=True,
-        ),
-        sa.ForeignKeyConstraint(
-            ["user_id"],
-            ["user.id"],
-        ),
-        sa.PrimaryKeyConstraint("id"),
-    )
-    op.create_table(
-        "inputprompt__user",
-        sa.Column("input_prompt_id", sa.Integer(), nullable=False),
-        sa.Column("user_id", sa.Integer(), nullable=False),
-        sa.ForeignKeyConstraint(
-            ["input_prompt_id"],
-            ["inputprompt.id"],
-        ),
-        sa.ForeignKeyConstraint(
-            ["user_id"],
-            ["inputprompt.id"],
-        ),
-        sa.PrimaryKeyConstraint("input_prompt_id", "user_id"),
-    )

backend/alembic/versions/f7a894b06d02_non_nullbale_slack_bot_id_in_channel_.py

@@ -1,40 +0,0 @@
-"""non-nullbale slack bot id in channel config
-
-Revision ID: f7a894b06d02
-Revises: 9f696734098f
-Create Date: 2024-12-06 12:55:42.845723
-
-"""
-
-from alembic import op
-import sqlalchemy as sa
-
-
-# revision identifiers, used by Alembic.
-revision = "f7a894b06d02"
-down_revision = "9f696734098f"
-branch_labels = None
-depends_on = None
-
-
-def upgrade() -> None:
-    # Delete all rows with null slack_bot_id
-    op.execute("DELETE FROM slack_channel_config WHERE slack_bot_id IS NULL")
-
-    # Make slack_bot_id non-nullable
-    op.alter_column(
-        "slack_channel_config",
-        "slack_bot_id",
-        existing_type=sa.Integer(),
-        nullable=False,
-    )
-
-
-def downgrade() -> None:
-    # Make slack_bot_id nullable again
-    op.alter_column(
-        "slack_channel_config",
-        "slack_bot_id",
-        existing_type=sa.Integer(),
-        nullable=True,
-    )

backend/alembic_tenants/env.py

@@ -1,6 +1,5 @@
 import asyncio
 from logging.config import fileConfig
-from typing import Literal
 
 from sqlalchemy import pool
 from sqlalchemy.engine import Connection
@@ -38,15 +37,8 @@ EXCLUDE_TABLES = {"kombu_queue", "kombu_message"}
 
 def include_object(
     object: SchemaItem,
-    name: str | None,
-    type_: Literal[
-        "schema",
-        "table",
-        "column",
-        "index",
-        "unique_constraint",
-        "foreign_key_constraint",
-    ],
+    name: str,
+    type_: str,
     reflected: bool,
     compare_to: SchemaItem | None,
 ) -> bool:

backend/danswer/access/models.py

@@ -18,11 +18,6 @@ class ExternalAccess:
 
 @dataclass(frozen=True)
 class DocExternalAccess:
-    """
-    This is just a class to wrap the external access and the document ID
-    together. It's used for syncing document permissions to Redis.
-    """
-
     external_access: ExternalAccess
     # The document ID
     doc_id: str

backend/danswer/auth/api_key.py

@@ -1,4 +1,3 @@
-import hashlib
 import secrets
 import uuid
 from urllib.parse import quote
@@ -19,8 +18,7 @@ _API_KEY_HEADER_NAME = "Authorization"
 # organizations like the Internet Engineering Task Force (IETF).
 _API_KEY_HEADER_ALTERNATIVE_NAME = "X-Danswer-Authorization"
 _BEARER_PREFIX = "Bearer "
-_API_KEY_PREFIX = "on_"
-_DEPRECATED_API_KEY_PREFIX = "dn_"
+_API_KEY_PREFIX = "dn_"
 _API_KEY_LEN = 192
 
 
@@ -54,9 +52,7 @@ def extract_tenant_from_api_key_header(request: Request) -> str | None:
 
     api_key = raw_api_key_header[len(_BEARER_PREFIX) :].strip()
 
-    if not api_key.startswith(_API_KEY_PREFIX) and not api_key.startswith(
-        _DEPRECATED_API_KEY_PREFIX
-    ):
+    if not api_key.startswith(_API_KEY_PREFIX):
         return None
 
     parts = api_key[len(_API_KEY_PREFIX) :].split(".", 1)
@@ -67,19 +63,10 @@ def extract_tenant_from_api_key_header(request: Request) -> str | None:
     return unquote(tenant_id) if tenant_id else None
 
 
-def _deprecated_hash_api_key(api_key: str) -> str:
-    return sha256_crypt.hash(api_key, salt="", rounds=API_KEY_HASH_ROUNDS)
-
-
 def hash_api_key(api_key: str) -> str:
     # NOTE: no salt is needed, as the API key is randomly generated
     # and overlaps are impossible
-    if api_key.startswith(_API_KEY_PREFIX):
-        return hashlib.sha256(api_key.encode("utf-8")).hexdigest()
-    elif api_key.startswith(_DEPRECATED_API_KEY_PREFIX):
-        return _deprecated_hash_api_key(api_key)
-    else:
-        raise ValueError(f"Invalid API key prefix: {api_key[:3]}")
+    return sha256_crypt.hash(api_key, salt="", rounds=API_KEY_HASH_ROUNDS)
 
 
 def build_displayable_api_key(api_key: str) -> str:

backend/danswer/auth/invited_users.py

@@ -9,6 +9,7 @@ from danswer.utils.special_types import JSON_ro
 def get_invited_users() -> list[str]:
     try:
         store = get_kv_store()
+
         return cast(list, store.load(KV_USER_STORE_KEY))
     except KvKeyNotFoundError:
         return list()

backend/danswer/auth/users.py

@@ -58,6 +58,7 @@ from danswer.auth.schemas import UserRole
 from danswer.auth.schemas import UserUpdate
 from danswer.configs.app_configs import AUTH_TYPE
 from danswer.configs.app_configs import DISABLE_AUTH
+from danswer.configs.app_configs import DISABLE_VERIFICATION
 from danswer.configs.app_configs import EMAIL_FROM
 from danswer.configs.app_configs import REQUIRE_EMAIL_VERIFICATION
 from danswer.configs.app_configs import SESSION_EXPIRE_TIME_SECONDS
@@ -86,7 +87,6 @@ from danswer.db.models import AccessToken
 from danswer.db.models import OAuthAccount
 from danswer.db.models import User
 from danswer.db.users import get_user_by_email
-from danswer.server.utils import BasicAuthenticationError
 from danswer.utils.logger import setup_logger
 from danswer.utils.telemetry import optional_telemetry
 from danswer.utils.telemetry import RecordType
@@ -99,6 +99,11 @@ from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
 logger = setup_logger()
 
 
+class BasicAuthenticationError(HTTPException):
+    def __init__(self, detail: str):
+        super().__init__(status_code=status.HTTP_403_FORBIDDEN, detail=detail)
+
+
 def is_user_admin(user: User | None) -> bool:
     if AUTH_TYPE == AuthType.DISABLED:
         return True
@@ -131,12 +136,11 @@ def get_display_email(email: str | None, space_less: bool = False) -> str:
 
 
 def user_needs_to_be_verified() -> bool:
-    if AUTH_TYPE == AuthType.BASIC or AUTH_TYPE == AuthType.CLOUD:
-        return REQUIRE_EMAIL_VERIFICATION
-
-    # For other auth types, if the user is authenticated it's assumed that
-    # the user is already verified via the external IDP
-    return False
+    # all other auth types besides basic should require users to be
+    # verified
+    return not DISABLE_VERIFICATION and (
+        AUTH_TYPE != AuthType.BASIC or REQUIRE_EMAIL_VERIFICATION
+    )
 
 
 def verify_email_is_invited(email: str) -> None:

backend/danswer/background/celery/apps/app_base.py

@@ -11,7 +11,6 @@ from celery.exceptions import WorkerShutdown
 from celery.states import READY_STATES
 from celery.utils.log import get_task_logger
 from celery.worker import strategy  # type: ignore
-from redis.lock import Lock as RedisLock
 from sentry_sdk.integrations.celery import CeleryIntegration
 from sqlalchemy import text
 from sqlalchemy.orm import Session
@@ -333,16 +332,16 @@ def on_worker_shutdown(sender: Any, **kwargs: Any) -> None:
         return
 
     logger.info("Releasing primary worker lock.")
-    lock: RedisLock = sender.primary_worker_lock
+    lock = sender.primary_worker_lock
     try:
         if lock.owned():
             try:
                 lock.release()
                 sender.primary_worker_lock = None
-            except Exception:
-                logger.exception("Failed to release primary worker lock")
-    except Exception:
-        logger.exception("Failed to check if primary worker lock is owned")
+            except Exception as e:
+                logger.error(f"Failed to release primary worker lock: {e}")
+    except Exception as e:
+        logger.error(f"Failed to check if primary worker lock is owned: {e}")
 
 
 def on_setup_logging(

backend/danswer/background/celery/apps/primary.py

@@ -11,7 +11,6 @@ from celery.signals import celeryd_init
 from celery.signals import worker_init
 from celery.signals import worker_ready
 from celery.signals import worker_shutdown
-from redis.lock import Lock as RedisLock
 
 import danswer.background.celery.apps.app_base as app_base
 from danswer.background.celery.apps.app_base import task_logger
@@ -39,6 +38,7 @@ from danswer.redis.redis_usergroup import RedisUserGroup
 from danswer.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 
+
 logger = setup_logger()
 
 celery_app = Celery(__name__)
@@ -116,13 +116,9 @@ def on_worker_init(sender: Any, **kwargs: Any) -> None:
     # it is planned to use this lock to enforce singleton behavior on the primary
     # worker, since the primary worker does redis cleanup on startup, but this isn't
     # implemented yet.
-
-    # set thread_local=False since we don't control what thread the periodic task might
-    # reacquire the lock with
-    lock: RedisLock = r.lock(
+    lock = r.lock(
         DanswerRedisLocks.PRIMARY_WORKER,
         timeout=CELERY_PRIMARY_WORKER_LOCK_TIMEOUT,
-        thread_local=False,
     )
 
     logger.info("Primary worker lock: Acquire starting.")
@@ -231,7 +227,7 @@ class HubPeriodicTask(bootsteps.StartStopStep):
             if not hasattr(worker, "primary_worker_lock"):
                 return
 
-            lock: RedisLock = worker.primary_worker_lock
+            lock = worker.primary_worker_lock
 
             r = get_redis_client(tenant_id=None)
 

backend/danswer/background/celery/tasks/beat_schedule.py

@@ -2,55 +2,54 @@ from datetime import timedelta
 from typing import Any
 
 from danswer.configs.constants import DanswerCeleryPriority
-from danswer.configs.constants import DanswerCeleryTask
 
 
 tasks_to_schedule = [
     {
         "name": "check-for-vespa-sync",
-        "task": DanswerCeleryTask.CHECK_FOR_VESPA_SYNC_TASK,
+        "task": "check_for_vespa_sync_task",
         "schedule": timedelta(seconds=20),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "check-for-connector-deletion",
-        "task": DanswerCeleryTask.CHECK_FOR_CONNECTOR_DELETION,
+        "task": "check_for_connector_deletion_task",
         "schedule": timedelta(seconds=20),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "check-for-indexing",
-        "task": DanswerCeleryTask.CHECK_FOR_INDEXING,
+        "task": "check_for_indexing",
         "schedule": timedelta(seconds=15),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "check-for-prune",
-        "task": DanswerCeleryTask.CHECK_FOR_PRUNING,
+        "task": "check_for_pruning",
         "schedule": timedelta(seconds=15),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "kombu-message-cleanup",
-        "task": DanswerCeleryTask.KOMBU_MESSAGE_CLEANUP_TASK,
+        "task": "kombu_message_cleanup_task",
         "schedule": timedelta(seconds=3600),
         "options": {"priority": DanswerCeleryPriority.LOWEST},
     },
     {
         "name": "monitor-vespa-sync",
-        "task": DanswerCeleryTask.MONITOR_VESPA_SYNC,
+        "task": "monitor_vespa_sync",
         "schedule": timedelta(seconds=5),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "check-for-doc-permissions-sync",
-        "task": DanswerCeleryTask.CHECK_FOR_DOC_PERMISSIONS_SYNC,
+        "task": "check_for_doc_permissions_sync",
         "schedule": timedelta(seconds=30),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },
     {
         "name": "check-for-external-group-sync",
-        "task": DanswerCeleryTask.CHECK_FOR_EXTERNAL_GROUP_SYNC,
+        "task": "check_for_external_group_sync",
         "schedule": timedelta(seconds=20),
         "options": {"priority": DanswerCeleryPriority.HIGH},
     },

backend/danswer/background/celery/tasks/connector_deletion/tasks.py

@@ -11,7 +11,6 @@ from sqlalchemy.orm import Session
 from danswer.background.celery.apps.app_base import task_logger
 from danswer.configs.app_configs import JOB_TIMEOUT
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
 from danswer.db.connector_credential_pair import get_connector_credential_pairs
@@ -29,7 +28,7 @@ class TaskDependencyError(RuntimeError):
 
 
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_CONNECTOR_DELETION,
+    name="check_for_connector_deletion_task",
     soft_time_limit=JOB_TIMEOUT,
     trail=False,
     bind=True,

backend/danswer/background/celery/tasks/doc_permission_syncing/tasks.py

@@ -18,11 +18,9 @@ from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.configs.constants import DocumentSource
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
-from danswer.db.document import upsert_document_by_connector_credential_pair
 from danswer.db.engine import get_session_with_tenant
 from danswer.db.enums import AccessType
 from danswer.db.enums import ConnectorCredentialPairStatus
@@ -84,7 +82,7 @@ def _is_external_doc_permissions_sync_due(cc_pair: ConnectorCredentialPair) -> b
 
 
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_DOC_PERMISSIONS_SYNC,
+    name="check_for_doc_permissions_sync",
     soft_time_limit=JOB_TIMEOUT,
     bind=True,
 )
@@ -166,7 +164,7 @@ def try_creating_permissions_sync_task(
         custom_task_id = f"{redis_connector.permissions.generator_task_key}_{uuid4()}"
 
         result = app.send_task(
-            DanswerCeleryTask.CONNECTOR_PERMISSION_SYNC_GENERATOR_TASK,
+            "connector_permission_sync_generator_task",
             kwargs=dict(
                 cc_pair_id=cc_pair_id,
                 tenant_id=tenant_id,
@@ -193,7 +191,7 @@ def try_creating_permissions_sync_task(
 
 
 @shared_task(
-    name=DanswerCeleryTask.CONNECTOR_PERMISSION_SYNC_GENERATOR_TASK,
+    name="connector_permission_sync_generator_task",
     acks_late=False,
     soft_time_limit=JOB_TIMEOUT,
     track_started=True,
@@ -219,7 +217,7 @@ def connector_permission_sync_generator_task(
 
     r = get_redis_client(tenant_id=tenant_id)
 
-    lock: RedisLock = r.lock(
+    lock = r.lock(
         DanswerRedisLocks.CONNECTOR_DOC_PERMISSIONS_SYNC_LOCK_PREFIX
         + f"_{redis_connector.id}",
         timeout=CELERY_PERMISSIONS_SYNC_LOCK_TIMEOUT,
@@ -263,12 +261,7 @@ def connector_permission_sync_generator_task(
                 f"RedisConnector.permissions.generate_tasks starting. cc_pair={cc_pair_id}"
             )
             tasks_generated = redis_connector.permissions.generate_tasks(
-                celery_app=self.app,
-                lock=lock,
-                new_permissions=document_external_accesses,
-                source_string=source_type,
-                connector_id=cc_pair.connector.id,
-                credential_id=cc_pair.credential.id,
+                self.app, lock, document_external_accesses, source_type
             )
             if tasks_generated is None:
                 return None
@@ -293,7 +286,7 @@ def connector_permission_sync_generator_task(
 
 
 @shared_task(
-    name=DanswerCeleryTask.UPDATE_EXTERNAL_DOCUMENT_PERMISSIONS_TASK,
+    name="update_external_document_permissions_task",
     soft_time_limit=LIGHT_SOFT_TIME_LIMIT,
     time_limit=LIGHT_TIME_LIMIT,
     max_retries=DOCUMENT_PERMISSIONS_UPDATE_MAX_RETRIES,
@@ -304,8 +297,6 @@ def update_external_document_permissions_task(
     tenant_id: str | None,
     serialized_doc_external_access: dict,
     source_string: str,
-    connector_id: int,
-    credential_id: int,
 ) -> bool:
     document_external_access = DocExternalAccess.from_dict(
         serialized_doc_external_access
@@ -314,28 +305,18 @@ def update_external_document_permissions_task(
     external_access = document_external_access.external_access
     try:
         with get_session_with_tenant(tenant_id) as db_session:
-            # Add the users to the DB if they don't exist
+            # Then we build the update requests to update vespa
             batch_add_ext_perm_user_if_not_exists(
                 db_session=db_session,
                 emails=list(external_access.external_user_emails),
             )
-            # Then we upsert the document's external permissions in postgres
-            created_new_doc = upsert_document_external_perms(
+            upsert_document_external_perms(
                 db_session=db_session,
                 doc_id=doc_id,
                 external_access=external_access,
                 source_type=DocumentSource(source_string),
             )
 
-            if created_new_doc:
-                # If a new document was created, we associate it with the cc_pair
-                upsert_document_by_connector_credential_pair(
-                    db_session=db_session,
-                    connector_id=connector_id,
-                    credential_id=credential_id,
-                    document_ids=[doc_id],
-                )
-
             logger.debug(
                 f"Successfully synced postgres document permissions for {doc_id}"
             )

backend/danswer/background/celery/tasks/external_group_syncing/tasks.py

@@ -17,7 +17,6 @@ from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.db.connector import mark_cc_pair_as_external_group_synced
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
@@ -32,14 +31,10 @@ from danswer.redis.redis_connector_ext_group_sync import (
 from danswer.redis.redis_pool import get_redis_client
 from danswer.utils.logger import setup_logger
 from ee.danswer.db.connector_credential_pair import get_all_auto_sync_cc_pairs
-from ee.danswer.db.connector_credential_pair import get_cc_pairs_by_source
 from ee.danswer.db.external_perm import ExternalUserGroup
 from ee.danswer.db.external_perm import replace_user__ext_group_for_cc_pair
 from ee.danswer.external_permissions.sync_params import EXTERNAL_GROUP_SYNC_PERIODS
 from ee.danswer.external_permissions.sync_params import GROUP_PERMISSIONS_FUNC_MAP
-from ee.danswer.external_permissions.sync_params import (
-    GROUP_PERMISSIONS_IS_CC_PAIR_AGNOSTIC,
-)
 
 logger = setup_logger()
 
@@ -90,7 +85,7 @@ def _is_external_group_sync_due(cc_pair: ConnectorCredentialPair) -> bool:
 
 
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_EXTERNAL_GROUP_SYNC,
+    name="check_for_external_group_sync",
     soft_time_limit=JOB_TIMEOUT,
     bind=True,
 )
@@ -111,22 +106,6 @@ def check_for_external_group_sync(self: Task, *, tenant_id: str | None) -> None:
         with get_session_with_tenant(tenant_id) as db_session:
             cc_pairs = get_all_auto_sync_cc_pairs(db_session)
 
-            # We only want to sync one cc_pair per source type in
-            # GROUP_PERMISSIONS_IS_CC_PAIR_AGNOSTIC
-            for source in GROUP_PERMISSIONS_IS_CC_PAIR_AGNOSTIC:
-                # These are ordered by cc_pair id so the first one is the one we want
-                cc_pairs_to_dedupe = get_cc_pairs_by_source(
-                    db_session, source, only_sync=True
-                )
-                # We only want to sync one cc_pair per source type
-                # in GROUP_PERMISSIONS_IS_CC_PAIR_AGNOSTIC so we dedupe here
-                for cc_pair_to_remove in cc_pairs_to_dedupe[1:]:
-                    cc_pairs = [
-                        cc_pair
-                        for cc_pair in cc_pairs
-                        if cc_pair.id != cc_pair_to_remove.id
-                    ]
-
             for cc_pair in cc_pairs:
                 if _is_external_group_sync_due(cc_pair):
                     cc_pair_ids_to_sync.append(cc_pair.id)
@@ -182,7 +161,7 @@ def try_creating_external_group_sync_task(
         custom_task_id = f"{redis_connector.external_group_sync.taskset_key}_{uuid4()}"
 
         result = app.send_task(
-            DanswerCeleryTask.CONNECTOR_EXTERNAL_GROUP_SYNC_GENERATOR_TASK,
+            "connector_external_group_sync_generator_task",
             kwargs=dict(
                 cc_pair_id=cc_pair_id,
                 tenant_id=tenant_id,
@@ -212,7 +191,7 @@ def try_creating_external_group_sync_task(
 
 
 @shared_task(
-    name=DanswerCeleryTask.CONNECTOR_EXTERNAL_GROUP_SYNC_GENERATOR_TASK,
+    name="connector_external_group_sync_generator_task",
     acks_late=False,
     soft_time_limit=JOB_TIMEOUT,
     track_started=True,

backend/danswer/background/celery/tasks/indexing/tasks.py

@@ -23,7 +23,6 @@ from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.configs.constants import DocumentSource
 from danswer.db.connector import mark_ccpair_with_indexing_trigger
@@ -157,7 +156,7 @@ def get_unfenced_index_attempt_ids(db_session: Session, r: redis.Redis) -> list[
 
 
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_INDEXING,
+    name="check_for_indexing",
     soft_time_limit=300,
     bind=True,
 )
@@ -487,7 +486,7 @@ def try_creating_indexing_task(
         # when the task is sent, we have yet to finish setting up the fence
         # therefore, the task must contain code that blocks until the fence is ready
         result = celery_app.send_task(
-            DanswerCeleryTask.CONNECTOR_INDEXING_PROXY_TASK,
+            "connector_indexing_proxy_task",
             kwargs=dict(
                 index_attempt_id=index_attempt_id,
                 cc_pair_id=cc_pair.id,
@@ -525,10 +524,7 @@ def try_creating_indexing_task(
 
 
 @shared_task(
-    name=DanswerCeleryTask.CONNECTOR_INDEXING_PROXY_TASK,
-    bind=True,
-    acks_late=False,
-    track_started=True,
+    name="connector_indexing_proxy_task", bind=True, acks_late=False, track_started=True
 )
 def connector_indexing_proxy_task(
     self: Task,
@@ -584,97 +580,45 @@ def connector_indexing_proxy_task(
 
         if self.request.id and redis_connector_index.terminating(self.request.id):
             task_logger.warning(
-                "Indexing watchdog - termination signal detected: "
+                "Indexing proxy - termination signal detected: "
                 f"attempt={index_attempt_id} "
                 f"tenant={tenant_id} "
                 f"cc_pair={cc_pair_id} "
                 f"search_settings={search_settings_id}"
             )
 
-            try:
-                with get_session_with_tenant(tenant_id) as db_session:
-                    mark_attempt_canceled(
-                        index_attempt_id,
-                        db_session,
-                        "Connector termination signal detected",
-                    )
-            except Exception:
-                # if the DB exceptions, we'll just get an unfriendly failure message
-                # in the UI instead of the cancellation message
-                logger.exception(
-                    "Indexing watchdog - transient exception marking index attempt as canceled: "
-                    f"attempt={index_attempt_id} "
-                    f"tenant={tenant_id} "
-                    f"cc_pair={cc_pair_id} "
-                    f"search_settings={search_settings_id}"
+            with get_session_with_tenant(tenant_id) as db_session:
+                mark_attempt_canceled(
+                    index_attempt_id,
+                    db_session,
+                    "Connector termination signal detected",
                 )
 
-                job.cancel()
-
+            job.cancel()
             break
 
+        # do nothing for ongoing jobs that haven't been stopped
         if not job.done():
-            # if the spawned task is still running, restart the check once again
-            # if the index attempt is not in a finished status
-            try:
-                with get_session_with_tenant(tenant_id) as db_session:
-                    index_attempt = get_index_attempt(
-                        db_session=db_session, index_attempt_id=index_attempt_id
-                    )
-
-                    if not index_attempt:
-                        continue
-
-                    if not index_attempt.is_finished():
-                        continue
-            except Exception:
-                # if the DB exceptioned, just restart the check.
-                # polling the index attempt status doesn't need to be strongly consistent
-                logger.exception(
-                    "Indexing watchdog - transient exception looking up index attempt: "
-                    f"attempt={index_attempt_id} "
-                    f"tenant={tenant_id} "
-                    f"cc_pair={cc_pair_id} "
-                    f"search_settings={search_settings_id}"
+            with get_session_with_tenant(tenant_id) as db_session:
+                index_attempt = get_index_attempt(
+                    db_session=db_session, index_attempt_id=index_attempt_id
                 )
-                continue
+
+                if not index_attempt:
+                    continue
+
+                if not index_attempt.is_finished():
+                    continue
 
         if job.status == "error":
-            ignore_exitcode = False
-
-            exit_code: int | None = None
-            if job.process:
-                exit_code = job.process.exitcode
-
-            # seeing non-deterministic behavior where spawned tasks occasionally return exit code 1
-            # even though logging clearly indicates that they completed successfully
-            # to work around this, we ignore the job error state if the completion signal is OK
-            status_int = redis_connector_index.get_completion()
-            if status_int:
-                status_enum = HTTPStatus(status_int)
-                if status_enum == HTTPStatus.OK:
-                    ignore_exitcode = True
-
-            if ignore_exitcode:
-                task_logger.warning(
-                    "Indexing watchdog - spawned task has non-zero exit code "
-                    "but completion signal is OK. Continuing...: "
-                    f"attempt={index_attempt_id} "
-                    f"tenant={tenant_id} "
-                    f"cc_pair={cc_pair_id} "
-                    f"search_settings={search_settings_id} "
-                    f"exit_code={exit_code}"
-                )
-            else:
-                task_logger.error(
-                    "Indexing watchdog - spawned task exceptioned: "
-                    f"attempt={index_attempt_id} "
-                    f"tenant={tenant_id} "
-                    f"cc_pair={cc_pair_id} "
-                    f"search_settings={search_settings_id} "
-                    f"exit_code={exit_code} "
-                    f"error={job.exception()}"
-                )
+            task_logger.error(
+                f"Indexing watchdog - spawned task exceptioned: "
+                f"attempt={index_attempt_id} "
+                f"tenant={tenant_id} "
+                f"cc_pair={cc_pair_id} "
+                f"search_settings={search_settings_id} "
+                f"error={job.exception()}"
+            )
 
         job.release()
         break
@@ -816,12 +760,9 @@ def connector_indexing_task(
         )
         break
 
-    # set thread_local=False since we don't control what thread the indexing/pruning
-    # might run our callback with
     lock: RedisLock = r.lock(
         redis_connector_index.generator_lock_key,
         timeout=CELERY_INDEXING_LOCK_TIMEOUT,
-        thread_local=False,
     )
 
     acquired = lock.acquire(blocking=False)

backend/danswer/background/celery/tasks/periodic/tasks.py

@@ -13,13 +13,12 @@ from sqlalchemy.orm import Session
 
 from danswer.background.celery.apps.app_base import task_logger
 from danswer.configs.app_configs import JOB_TIMEOUT
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import PostgresAdvisoryLocks
 from danswer.db.engine import get_session_with_tenant
 
 
 @shared_task(
-    name=DanswerCeleryTask.KOMBU_MESSAGE_CLEANUP_TASK,
+    name="kombu_message_cleanup_task",
     soft_time_limit=JOB_TIMEOUT,
     bind=True,
     base=AbortableTask,

backend/danswer/background/celery/tasks/pruning/tasks.py

@@ -8,7 +8,6 @@ from celery import shared_task
 from celery import Task
 from celery.exceptions import SoftTimeLimitExceeded
 from redis import Redis
-from redis.lock import Lock as RedisLock
 from sqlalchemy.orm import Session
 
 from danswer.background.celery.apps.app_base import task_logger
@@ -21,7 +20,6 @@ from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DANSWER_REDIS_FUNCTION_LOCK_PREFIX
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.connectors.factory import instantiate_connector
 from danswer.connectors.models import InputType
@@ -77,7 +75,7 @@ def _is_pruning_due(cc_pair: ConnectorCredentialPair) -> bool:
 
 
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_PRUNING,
+    name="check_for_pruning",
     soft_time_limit=JOB_TIMEOUT,
     bind=True,
 )
@@ -186,7 +184,7 @@ def try_creating_prune_generator_task(
         custom_task_id = f"{redis_connector.prune.generator_task_key}_{uuid4()}"
 
         celery_app.send_task(
-            DanswerCeleryTask.CONNECTOR_PRUNING_GENERATOR_TASK,
+            "connector_pruning_generator_task",
             kwargs=dict(
                 cc_pair_id=cc_pair.id,
                 connector_id=cc_pair.connector_id,
@@ -211,7 +209,7 @@ def try_creating_prune_generator_task(
 
 
 @shared_task(
-    name=DanswerCeleryTask.CONNECTOR_PRUNING_GENERATOR_TASK,
+    name="connector_pruning_generator_task",
     acks_late=False,
     soft_time_limit=JOB_TIMEOUT,
     track_started=True,
@@ -240,12 +238,9 @@ def connector_pruning_generator_task(
 
     r = get_redis_client(tenant_id=tenant_id)
 
-    # set thread_local=False since we don't control what thread the indexing/pruning
-    # might run our callback with
-    lock: RedisLock = r.lock(
+    lock = r.lock(
         DanswerRedisLocks.PRUNING_LOCK_PREFIX + f"_{redis_connector.id}",
         timeout=CELERY_PRUNING_LOCK_TIMEOUT,
-        thread_local=False,
     )
 
     acquired = lock.acquire(blocking=False)

backend/danswer/background/celery/tasks/shared/tasks.py

@@ -9,7 +9,6 @@ from tenacity import RetryError
 from danswer.access.access import get_access_for_document
 from danswer.background.celery.apps.app_base import task_logger
 from danswer.background.celery.tasks.shared.RetryDocumentIndex import RetryDocumentIndex
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.db.document import delete_document_by_connector_credential_pair__no_commit
 from danswer.db.document import delete_documents_complete__no_commit
 from danswer.db.document import get_document
@@ -32,7 +31,7 @@ LIGHT_TIME_LIMIT = LIGHT_SOFT_TIME_LIMIT + 15
 
 
 @shared_task(
-    name=DanswerCeleryTask.DOCUMENT_BY_CC_PAIR_CLEANUP_TASK,
+    name="document_by_cc_pair_cleanup_task",
     soft_time_limit=LIGHT_SOFT_TIME_LIMIT,
     time_limit=LIGHT_TIME_LIMIT,
     max_retries=DOCUMENT_BY_CC_PAIR_CLEANUP_MAX_RETRIES,

backend/danswer/background/celery/tasks/vespa/tasks.py

@@ -25,7 +25,6 @@ from danswer.background.celery.tasks.shared.tasks import LIGHT_TIME_LIMIT
 from danswer.configs.app_configs import JOB_TIMEOUT
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.configs.constants import DanswerRedisLocks
 from danswer.db.connector import fetch_connector_by_id
 from danswer.db.connector import mark_cc_pair_as_permissions_synced
@@ -81,7 +80,7 @@ logger = setup_logger()
 # celery auto associates tasks created inside another task,
 # which bloats the result metadata considerably. trail=False prevents this.
 @shared_task(
-    name=DanswerCeleryTask.CHECK_FOR_VESPA_SYNC_TASK,
+    name="check_for_vespa_sync_task",
     soft_time_limit=JOB_TIMEOUT,
     trail=False,
     bind=True,
@@ -655,53 +654,38 @@ def monitor_ccpair_indexing_taskset(
     # outer = result.state in READY state
     status_int = redis_connector_index.get_completion()
     if status_int is None:  # inner signal not set ... possible error
-        task_state = result.state
+        result_state = result.state
         if (
-            task_state in READY_STATES
+            result_state in READY_STATES
         ):  # outer signal in terminal state ... possible error
             # Now double check!
             if redis_connector_index.get_completion() is None:
                 # inner signal still not set (and cannot change when outer result_state is READY)
                 # Task is finished but generator complete isn't set.
                 # We have a problem! Worker may have crashed.
-                task_result = str(result.result)
-                task_traceback = str(result.traceback)
 
                 msg = (
                     f"Connector indexing aborted or exceptioned: "
                     f"attempt={payload.index_attempt_id} "
                     f"celery_task={payload.celery_task_id} "
+                    f"result_state={result_state} "
                     f"cc_pair={cc_pair_id} "
                     f"search_settings={search_settings_id} "
-                    f"elapsed_submitted={elapsed_submitted.total_seconds():.2f} "
-                    f"result.state={task_state} "
-                    f"result.result={task_result} "
-                    f"result.traceback={task_traceback}"
+                    f"elapsed_submitted={elapsed_submitted.total_seconds():.2f}"
                 )
                 task_logger.warning(msg)
 
-                try:
-                    index_attempt = get_index_attempt(
-                        db_session, payload.index_attempt_id
-                    )
-                    if index_attempt:
-                        if (
-                            index_attempt.status != IndexingStatus.CANCELED
-                            and index_attempt.status != IndexingStatus.FAILED
-                        ):
-                            mark_attempt_failed(
-                                index_attempt_id=payload.index_attempt_id,
-                                db_session=db_session,
-                                failure_reason=msg,
-                            )
-                except Exception:
-                    task_logger.exception(
-                        "monitor_ccpair_indexing_taskset - transient exception marking index attempt as failed: "
-                        f"attempt={payload.index_attempt_id} "
-                        f"tenant={tenant_id} "
-                        f"cc_pair={cc_pair_id} "
-                        f"search_settings={search_settings_id}"
-                    )
+                index_attempt = get_index_attempt(db_session, payload.index_attempt_id)
+                if index_attempt:
+                    if (
+                        index_attempt.status != IndexingStatus.CANCELED
+                        and index_attempt.status != IndexingStatus.FAILED
+                    ):
+                        mark_attempt_failed(
+                            index_attempt_id=payload.index_attempt_id,
+                            db_session=db_session,
+                            failure_reason=msg,
+                        )
 
                 redis_connector_index.reset()
         return
@@ -719,7 +703,7 @@ def monitor_ccpair_indexing_taskset(
     redis_connector_index.reset()
 
 
-@shared_task(name=DanswerCeleryTask.MONITOR_VESPA_SYNC, soft_time_limit=300, bind=True)
+@shared_task(name="monitor_vespa_sync", soft_time_limit=300, bind=True)
 def monitor_vespa_sync(self: Task, tenant_id: str | None) -> bool:
     """This is a celery beat task that monitors and finalizes metadata sync tasksets.
     It scans for fence values and then gets the counts of any associated tasksets.
@@ -830,7 +814,7 @@ def monitor_vespa_sync(self: Task, tenant_id: str | None) -> bool:
 
 
 @shared_task(
-    name=DanswerCeleryTask.VESPA_METADATA_SYNC_TASK,
+    name="vespa_metadata_sync_task",
     bind=True,
     soft_time_limit=LIGHT_SOFT_TIME_LIMIT,
     time_limit=LIGHT_TIME_LIMIT,

backend/danswer/background/indexing/job_client.py

@@ -82,7 +82,7 @@ class SimpleJob:
             return "running"
         elif self.process.exitcode is None:
             return "cancelled"
-        elif self.process.exitcode != 0:
+        elif self.process.exitcode > 0:
             return "error"
         else:
             return "finished"
@@ -123,8 +123,7 @@ class SimpleJobClient:
         self._cleanup_completed_jobs()
         if len(self.jobs) >= self.n_workers:
             logger.debug(
-                f"No available workers to run job. "
-                f"Currently running '{len(self.jobs)}' jobs, with a limit of '{self.n_workers}'."
+                f"No available workers to run job. Currently running '{len(self.jobs)}' jobs, with a limit of '{self.n_workers}'."
             )
             return None
 

backend/danswer/chat/chat_utils.py

@@ -2,79 +2,20 @@ import re
 from typing import cast
 from uuid import UUID
 
-from fastapi import HTTPException
 from fastapi.datastructures import Headers
 from sqlalchemy.orm import Session
 
-from danswer.auth.users import is_user_admin
 from danswer.chat.models import CitationInfo
 from danswer.chat.models import LlmDoc
-from danswer.chat.models import PersonaOverrideConfig
-from danswer.chat.models import ThreadMessage
-from danswer.configs.constants import DEFAULT_PERSONA_ID
-from danswer.configs.constants import MessageType
 from danswer.context.search.models import InferenceSection
-from danswer.context.search.models import RerankingDetails
-from danswer.context.search.models import RetrievalDetails
-from danswer.db.chat import create_chat_session
 from danswer.db.chat import get_chat_messages_by_session
-from danswer.db.llm import fetch_existing_doc_sets
-from danswer.db.llm import fetch_existing_tools
 from danswer.db.models import ChatMessage
-from danswer.db.models import Persona
-from danswer.db.models import Prompt
-from danswer.db.models import Tool
-from danswer.db.models import User
-from danswer.db.persona import get_prompts_by_ids
-from danswer.llm.models import PreviousMessage
-from danswer.natural_language_processing.utils import BaseTokenizer
-from danswer.server.query_and_chat.models import CreateChatMessageRequest
-from danswer.tools.tool_implementations.custom.custom_tool import (
-    build_custom_tools_from_openapi_schema_and_headers,
-)
+from danswer.llm.answering.models import PreviousMessage
 from danswer.utils.logger import setup_logger
 
 logger = setup_logger()
 
 
-def prepare_chat_message_request(
-    message_text: str,
-    user: User | None,
-    persona_id: int | None,
-    # Does the question need to have a persona override
-    persona_override_config: PersonaOverrideConfig | None,
-    prompt: Prompt | None,
-    message_ts_to_respond_to: str | None,
-    retrieval_details: RetrievalDetails | None,
-    rerank_settings: RerankingDetails | None,
-    db_session: Session,
-) -> CreateChatMessageRequest:
-    # Typically used for one shot flows like SlackBot or non-chat API endpoint use cases
-    new_chat_session = create_chat_session(
-        db_session=db_session,
-        description=None,
-        user_id=user.id if user else None,
-        # If using an override, this id will be ignored later on
-        persona_id=persona_id or DEFAULT_PERSONA_ID,
-        danswerbot_flow=True,
-        slack_thread_id=message_ts_to_respond_to,
-    )
-
-    return CreateChatMessageRequest(
-        chat_session_id=new_chat_session.id,
-        parent_message_id=None,  # It's a standalone chat session each time
-        message=message_text,
-        file_descriptors=[],  # Currently SlackBot/answer api do not support files in the context
-        prompt_id=prompt.id if prompt else None,
-        # Can always override the persona for the single query, if it's a normal persona
-        # then it will be treated the same
-        persona_override_config=persona_override_config,
-        search_doc_ids=None,
-        retrieval_options=retrieval_details,
-        rerank_settings=rerank_settings,
-    )
-
-
 def llm_doc_from_inference_section(inference_section: InferenceSection) -> LlmDoc:
     return LlmDoc(
         document_id=inference_section.center_chunk.document_id,
@@ -90,49 +31,9 @@ def llm_doc_from_inference_section(inference_section: InferenceSection) -> LlmDo
         if inference_section.center_chunk.source_links
         else None,
         source_links=inference_section.center_chunk.source_links,
-        match_highlights=inference_section.center_chunk.match_highlights,
     )
 
 
-def combine_message_thread(
-    messages: list[ThreadMessage],
-    max_tokens: int | None,
-    llm_tokenizer: BaseTokenizer,
-) -> str:
-    """Used to create a single combined message context from threads"""
-    if not messages:
-        return ""
-
-    message_strs: list[str] = []
-    total_token_count = 0
-
-    for message in reversed(messages):
-        if message.role == MessageType.USER:
-            role_str = message.role.value.upper()
-            if message.sender:
-                role_str += " " + message.sender
-            else:
-                # Since other messages might have the user identifying information
-                # better to use Unknown for symmetry
-                role_str += " Unknown"
-        else:
-            role_str = message.role.value.upper()
-
-        msg_str = f"{role_str}:\n{message.message}"
-        message_token_count = len(llm_tokenizer.encode(msg_str))
-
-        if (
-            max_tokens is not None
-            and total_token_count + message_token_count > max_tokens
-        ):
-            break
-
-        message_strs.insert(0, msg_str)
-        total_token_count += message_token_count
-
-    return "\n\n".join(message_strs)
-
-
 def create_chat_chain(
     chat_session_id: UUID,
     db_session: Session,
@@ -295,71 +196,3 @@ def extract_headers(
             if lowercase_key in headers:
                 extracted_headers[lowercase_key] = headers[lowercase_key]
     return extracted_headers
-
-
-def create_temporary_persona(
-    persona_config: PersonaOverrideConfig, db_session: Session, user: User | None = None
-) -> Persona:
-    if not is_user_admin(user):
-        raise HTTPException(
-            status_code=403,
-            detail="User is not authorized to create a persona in one shot queries",
-        )
-
-    """Create a temporary Persona object from the provided configuration."""
-    persona = Persona(
-        name=persona_config.name,
-        description=persona_config.description,
-        num_chunks=persona_config.num_chunks,
-        llm_relevance_filter=persona_config.llm_relevance_filter,
-        llm_filter_extraction=persona_config.llm_filter_extraction,
-        recency_bias=persona_config.recency_bias,
-        llm_model_provider_override=persona_config.llm_model_provider_override,
-        llm_model_version_override=persona_config.llm_model_version_override,
-    )
-
-    if persona_config.prompts:
-        persona.prompts = [
-            Prompt(
-                name=p.name,
-                description=p.description,
-                system_prompt=p.system_prompt,
-                task_prompt=p.task_prompt,
-                include_citations=p.include_citations,
-                datetime_aware=p.datetime_aware,
-            )
-            for p in persona_config.prompts
-        ]
-    elif persona_config.prompt_ids:
-        persona.prompts = get_prompts_by_ids(
-            db_session=db_session, prompt_ids=persona_config.prompt_ids
-        )
-
-    persona.tools = []
-    if persona_config.custom_tools_openapi:
-        for schema in persona_config.custom_tools_openapi:
-            tools = cast(
-                list[Tool],
-                build_custom_tools_from_openapi_schema_and_headers(schema),
-            )
-            persona.tools.extend(tools)
-
-    if persona_config.tools:
-        tool_ids = [tool.id for tool in persona_config.tools]
-        persona.tools.extend(
-            fetch_existing_tools(db_session=db_session, tool_ids=tool_ids)
-        )
-
-    if persona_config.tool_ids:
-        persona.tools.extend(
-            fetch_existing_tools(
-                db_session=db_session, tool_ids=persona_config.tool_ids
-            )
-        )
-
-    fetched_docs = fetch_existing_doc_sets(
-        db_session=db_session, doc_ids=persona_config.document_set_ids
-    )
-    persona.document_sets = fetched_docs
-
-    return persona

backend/danswer/chat/models.py

@@ -1,30 +1,17 @@
-from collections.abc import Callable
 from collections.abc import Iterator
 from datetime import datetime
 from enum import Enum
 from typing import Any
-from typing import TYPE_CHECKING
 
 from pydantic import BaseModel
-from pydantic import ConfigDict
-from pydantic import Field
-from pydantic import model_validator
 
 from danswer.configs.constants import DocumentSource
-from danswer.configs.constants import MessageType
 from danswer.context.search.enums import QueryFlow
-from danswer.context.search.enums import RecencyBiasSetting
 from danswer.context.search.enums import SearchType
 from danswer.context.search.models import RetrievalDocs
-from danswer.llm.override_models import PromptOverride
-from danswer.tools.models import ToolCallFinalResult
-from danswer.tools.models import ToolCallKickoff
-from danswer.tools.models import ToolResponse
+from danswer.context.search.models import SearchResponse
 from danswer.tools.tool_implementations.custom.base_tool_types import ToolResultType
 
-if TYPE_CHECKING:
-    from danswer.db.models import Prompt
-
 
 class LlmDoc(BaseModel):
     """This contains the minimal set information for the LLM portion including citations"""
@@ -38,7 +25,6 @@ class LlmDoc(BaseModel):
     updated_at: datetime | None
     link: str | None
     source_links: dict[int, str] | None
-    match_highlights: list[str] | None
 
 
 # First chunk of info for streaming QA
@@ -131,6 +117,20 @@ class StreamingError(BaseModel):
     stack_trace: str | None = None
 
 
+class DanswerQuote(BaseModel):
+    # This is during inference so everything is a string by this point
+    quote: str
+    document_id: str
+    link: str | None
+    source_type: str
+    semantic_identifier: str
+    blurb: str
+
+
+class DanswerQuotes(BaseModel):
+    quotes: list[DanswerQuote]
+
+
 class DanswerContext(BaseModel):
     content: str
     document_id: str
@@ -146,20 +146,14 @@ class DanswerAnswer(BaseModel):
     answer: str | None
 
 
-class ThreadMessage(BaseModel):
-    message: str
-    sender: str | None = None
-    role: MessageType = MessageType.USER
-
-
-class ChatDanswerBotResponse(BaseModel):
-    answer: str | None = None
-    citations: list[CitationInfo] | None = None
-    docs: QADocsResponse | None = None
+class QAResponse(SearchResponse, DanswerAnswer):
+    quotes: list[DanswerQuote] | None
+    contexts: list[DanswerContexts] | None
+    predicted_flow: QueryFlow
+    predicted_search: SearchType
+    eval_res_valid: bool | None = None
     llm_selected_doc_indices: list[int] | None = None
     error_msg: str | None = None
-    chat_message_id: int | None = None
-    answer_valid: bool = True  # Reflexion result, default True if Reflexion not run
 
 
 class FileChatDisplay(BaseModel):
@@ -171,41 +165,9 @@ class CustomToolResponse(BaseModel):
     tool_name: str
 
 
-class ToolConfig(BaseModel):
-    id: int
-
-
-class PromptOverrideConfig(BaseModel):
-    name: str
-    description: str = ""
-    system_prompt: str
-    task_prompt: str = ""
-    include_citations: bool = True
-    datetime_aware: bool = True
-
-
-class PersonaOverrideConfig(BaseModel):
-    name: str
-    description: str
-    search_type: SearchType = SearchType.SEMANTIC
-    num_chunks: float | None = None
-    llm_relevance_filter: bool = False
-    llm_filter_extraction: bool = False
-    recency_bias: RecencyBiasSetting = RecencyBiasSetting.AUTO
-    llm_model_provider_override: str | None = None
-    llm_model_version_override: str | None = None
-
-    prompts: list[PromptOverrideConfig] = Field(default_factory=list)
-    prompt_ids: list[int] = Field(default_factory=list)
-
-    document_set_ids: list[int] = Field(default_factory=list)
-    tools: list[ToolConfig] = Field(default_factory=list)
-    tool_ids: list[int] = Field(default_factory=list)
-    custom_tools_openapi: list[dict[str, Any]] = Field(default_factory=list)
-
-
 AnswerQuestionPossibleReturn = (
     DanswerAnswerPiece
+    | DanswerQuotes
     | CitationInfo
     | DanswerContexts
     | FileChatDisplay
@@ -221,109 +183,3 @@ AnswerQuestionStreamReturn = Iterator[AnswerQuestionPossibleReturn]
 class LLMMetricsContainer(BaseModel):
     prompt_tokens: int
     response_tokens: int
-
-
-StreamProcessor = Callable[[Iterator[str]], AnswerQuestionStreamReturn]
-
-
-class DocumentPruningConfig(BaseModel):
-    max_chunks: int | None = None
-    max_window_percentage: float | None = None
-    max_tokens: int | None = None
-    # different pruning behavior is expected when the
-    # user manually selects documents they want to chat with
-    # e.g. we don't want to truncate each document to be no more
-    # than one chunk long
-    is_manually_selected_docs: bool = False
-    # If user specifies to include additional context Chunks for each match, then different pruning
-    # is used. As many Sections as possible are included, and the last Section is truncated
-    # If this is false, all of the Sections are truncated if they are longer than the expected Chunk size.
-    # Sections are often expected to be longer than the maximum Chunk size but Chunks should not be.
-    use_sections: bool = True
-    # If using tools, then we need to consider the tool length
-    tool_num_tokens: int = 0
-    # If using a tool message to represent the docs, then we have to JSON serialize
-    # the document content, which adds to the token count.
-    using_tool_message: bool = False
-
-
-class ContextualPruningConfig(DocumentPruningConfig):
-    num_chunk_multiple: int
-
-    @classmethod
-    def from_doc_pruning_config(
-        cls, num_chunk_multiple: int, doc_pruning_config: DocumentPruningConfig
-    ) -> "ContextualPruningConfig":
-        return cls(num_chunk_multiple=num_chunk_multiple, **doc_pruning_config.dict())
-
-
-class CitationConfig(BaseModel):
-    all_docs_useful: bool = False
-
-
-class QuotesConfig(BaseModel):
-    pass
-
-
-class AnswerStyleConfig(BaseModel):
-    citation_config: CitationConfig | None = None
-    quotes_config: QuotesConfig | None = None
-    document_pruning_config: DocumentPruningConfig = Field(
-        default_factory=DocumentPruningConfig
-    )
-    # forces the LLM to return a structured response, see
-    # https://platform.openai.com/docs/guides/structured-outputs/introduction
-    # right now, only used by the simple chat API
-    structured_response_format: dict | None = None
-
-    @model_validator(mode="after")
-    def check_quotes_and_citation(self) -> "AnswerStyleConfig":
-        if self.citation_config is None and self.quotes_config is None:
-            raise ValueError(
-                "One of `citation_config` or `quotes_config` must be provided"
-            )
-
-        if self.citation_config is not None and self.quotes_config is not None:
-            raise ValueError(
-                "Only one of `citation_config` or `quotes_config` must be provided"
-            )
-
-        return self
-
-
-class PromptConfig(BaseModel):
-    """Final representation of the Prompt configuration passed
-    into the `Answer` object."""
-
-    system_prompt: str
-    task_prompt: str
-    datetime_aware: bool
-    include_citations: bool
-
-    @classmethod
-    def from_model(
-        cls, model: "Prompt", prompt_override: PromptOverride | None = None
-    ) -> "PromptConfig":
-        override_system_prompt = (
-            prompt_override.system_prompt if prompt_override else None
-        )
-        override_task_prompt = prompt_override.task_prompt if prompt_override else None
-
-        return cls(
-            system_prompt=override_system_prompt or model.system_prompt,
-            task_prompt=override_task_prompt or model.task_prompt,
-            datetime_aware=model.datetime_aware,
-            include_citations=model.include_citations,
-        )
-
-    model_config = ConfigDict(frozen=True)
-
-
-ResponsePart = (
-    DanswerAnswerPiece
-    | CitationInfo
-    | ToolCallKickoff
-    | ToolResponse
-    | ToolCallFinalResult
-    | StreamStopInfo
-)

backend/danswer/chat/process_message.py

@@ -6,24 +6,16 @@ from typing import cast
 
 from sqlalchemy.orm import Session
 
-from danswer.chat.answer import Answer
 from danswer.chat.chat_utils import create_chat_chain
-from danswer.chat.chat_utils import create_temporary_persona
 from danswer.chat.models import AllCitations
-from danswer.chat.models import AnswerStyleConfig
-from danswer.chat.models import ChatDanswerBotResponse
-from danswer.chat.models import CitationConfig
 from danswer.chat.models import CitationInfo
 from danswer.chat.models import CustomToolResponse
 from danswer.chat.models import DanswerAnswerPiece
-from danswer.chat.models import DanswerContexts
-from danswer.chat.models import DocumentPruningConfig
 from danswer.chat.models import FileChatDisplay
 from danswer.chat.models import FinalUsedContextDocsResponse
 from danswer.chat.models import LLMRelevanceFilterResponse
 from danswer.chat.models import MessageResponseIDInfo
 from danswer.chat.models import MessageSpecificCitations
-from danswer.chat.models import PromptConfig
 from danswer.chat.models import QADocsResponse
 from danswer.chat.models import StreamingError
 from danswer.chat.models import StreamStopInfo
@@ -62,11 +54,16 @@ from danswer.document_index.factory import get_default_document_index
 from danswer.file_store.models import ChatFileType
 from danswer.file_store.models import FileDescriptor
 from danswer.file_store.utils import load_all_chat_files
-from danswer.file_store.utils import save_files
+from danswer.file_store.utils import save_files_from_urls
+from danswer.llm.answering.answer import Answer
+from danswer.llm.answering.models import AnswerStyleConfig
+from danswer.llm.answering.models import CitationConfig
+from danswer.llm.answering.models import DocumentPruningConfig
+from danswer.llm.answering.models import PreviousMessage
+from danswer.llm.answering.models import PromptConfig
 from danswer.llm.exceptions import GenAIDisabledException
 from danswer.llm.factory import get_llms_for_persona
 from danswer.llm.factory import get_main_llm_from_tuple
-from danswer.llm.models import PreviousMessage
 from danswer.llm.utils import litellm_exception_to_error_msg
 from danswer.natural_language_processing.utils import get_tokenizer
 from danswer.server.query_and_chat.models import ChatMessageDetail
@@ -105,7 +102,6 @@ from danswer.tools.tool_implementations.internet_search.internet_search_tool imp
 from danswer.tools.tool_implementations.search.search_tool import (
     FINAL_CONTEXT_DOCUMENTS_ID,
 )
-from danswer.tools.tool_implementations.search.search_tool import SEARCH_DOC_CONTENT_ID
 from danswer.tools.tool_implementations.search.search_tool import (
     SEARCH_RESPONSE_SUMMARY_ID,
 )
@@ -117,10 +113,7 @@ from danswer.tools.tool_implementations.search.search_tool import (
 from danswer.tools.tool_runner import ToolCallFinalResult
 from danswer.utils.logger import setup_logger
 from danswer.utils.long_term_log import LongTermLogger
-from danswer.utils.timing import log_function_time
 from danswer.utils.timing import log_generator_function_time
-from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
-
 
 logger = setup_logger()
 
@@ -263,7 +256,6 @@ def _get_force_search_settings(
 ChatPacket = (
     StreamingError
     | QADocsResponse
-    | DanswerContexts
     | LLMRelevanceFilterResponse
     | FinalUsedContextDocsResponse
     | ChatMessageDetail
@@ -294,8 +286,6 @@ def stream_chat_message_objects(
     custom_tool_additional_headers: dict[str, str] | None = None,
     is_connected: Callable[[], bool] | None = None,
     enforce_chat_session_id_for_search_docs: bool = True,
-    bypass_acl: bool = False,
-    include_contexts: bool = False,
 ) -> ChatPacketStream:
     """Streams in order:
     1. [conditional] Retrieved documents if a search needs to be run
@@ -303,7 +293,6 @@ def stream_chat_message_objects(
     3. [always] A set of streamed LLM tokens or an error anywhere along the line if something fails
     4. [always] Details on the final AI response message that is created
     """
-    tenant_id = CURRENT_TENANT_ID_CONTEXTVAR.get()
     use_existing_user_message = new_msg_req.use_existing_user_message
     existing_assistant_message_id = new_msg_req.existing_assistant_message_id
 
@@ -333,31 +322,17 @@ def stream_chat_message_objects(
             metadata={"user_id": str(user_id), "chat_session_id": str(chat_session_id)}
         )
 
+        # use alternate persona if alternative assistant id is passed in
         if alternate_assistant_id is not None:
-            # Allows users to specify a temporary persona (assistant) in the chat session
-            # this takes highest priority since it's user specified
             persona = get_persona_by_id(
                 alternate_assistant_id,
                 user=user,
                 db_session=db_session,
                 is_for_edit=False,
             )
-        elif new_msg_req.persona_override_config:
-            # Certain endpoints allow users to specify arbitrary persona settings
-            # this should never conflict with the alternate_assistant_id
-            persona = persona = create_temporary_persona(
-                db_session=db_session,
-                persona_config=new_msg_req.persona_override_config,
-                user=user,
-            )
         else:
             persona = chat_session.persona
 
-        if not persona:
-            raise RuntimeError("No persona specified or found for chat session")
-
-        # If a prompt override is specified via the API, use that with highest priority
-        # but for saving it, we are just mapping it to an existing prompt
         prompt_id = new_msg_req.prompt_id
         if prompt_id is None and persona.prompts:
             prompt_id = sorted(persona.prompts, key=lambda x: x.id)[-1].id
@@ -580,34 +555,19 @@ def stream_chat_message_objects(
             reserved_message_id=reserved_message_id,
         )
 
-        prompt_override = new_msg_req.prompt_override or chat_session.prompt_override
-        if new_msg_req.persona_override_config:
-            prompt_config = PromptConfig(
-                system_prompt=new_msg_req.persona_override_config.prompts[
-                    0
-                ].system_prompt,
-                task_prompt=new_msg_req.persona_override_config.prompts[0].task_prompt,
-                datetime_aware=new_msg_req.persona_override_config.prompts[
-                    0
-                ].datetime_aware,
-                include_citations=new_msg_req.persona_override_config.prompts[
-                    0
-                ].include_citations,
-            )
-        elif prompt_override:
-            if not final_msg.prompt:
-                raise ValueError(
-                    "Prompt override cannot be applied, no base prompt found."
-                )
-            prompt_config = PromptConfig.from_model(
-                final_msg.prompt,
-                prompt_override=prompt_override,
-            )
-        elif final_msg.prompt:
-            prompt_config = PromptConfig.from_model(final_msg.prompt)
-        else:
-            prompt_config = PromptConfig.from_model(persona.prompts[0])
+        if not final_msg.prompt:
+            raise RuntimeError("No Prompt found")
 
+        prompt_config = (
+            PromptConfig.from_model(
+                final_msg.prompt,
+                prompt_override=(
+                    new_msg_req.prompt_override or chat_session.prompt_override
+                ),
+            )
+            if not persona
+            else PromptConfig.from_model(persona.prompts[0])
+        )
         answer_style_config = AnswerStyleConfig(
             citation_config=CitationConfig(
                 all_docs_useful=selected_db_search_docs is not None
@@ -627,13 +587,11 @@ def stream_chat_message_objects(
                 answer_style_config=answer_style_config,
                 document_pruning_config=document_pruning_config,
                 retrieval_options=retrieval_options or RetrievalDetails(),
-                rerank_settings=new_msg_req.rerank_settings,
                 selected_sections=selected_sections,
                 chunks_above=new_msg_req.chunks_above,
                 chunks_below=new_msg_req.chunks_below,
                 full_doc=new_msg_req.full_doc,
                 latest_query_files=latest_query_files,
-                bypass_acl=bypass_acl,
             ),
             internet_search_tool_config=InternetSearchToolConfig(
                 answer_style_config=answer_style_config,
@@ -680,8 +638,7 @@ def stream_chat_message_objects(
 
         reference_db_search_docs = None
         qa_docs_response = None
-        # any files to associate with the AI message e.g. dall-e generated images
-        ai_message_files = []
+        ai_message_files = None  # any files to associate with the AI message e.g. dall-e generated images
         dropped_indices = None
         tool_result = None
 
@@ -736,14 +693,8 @@ def stream_chat_message_objects(
                         list[ImageGenerationResponse], packet.response
                     )
 
-                    file_ids = save_files(
-                        urls=[img.url for img in img_generation_response if img.url],
-                        base64_files=[
-                            img.image_data
-                            for img in img_generation_response
-                            if img.image_data
-                        ],
-                        tenant_id=tenant_id,
+                    file_ids = save_files_from_urls(
+                        [img.url for img in img_generation_response]
                     )
                     ai_message_files = [
                         FileDescriptor(id=str(file_id), type=ChatFileType.IMAGE)
@@ -769,19 +720,15 @@ def stream_chat_message_objects(
                         or custom_tool_response.response_type == "csv"
                     ):
                         file_ids = custom_tool_response.tool_result.file_ids
-                        ai_message_files.extend(
-                            [
-                                FileDescriptor(
-                                    id=str(file_id),
-                                    type=(
-                                        ChatFileType.IMAGE
-                                        if custom_tool_response.response_type == "image"
-                                        else ChatFileType.CSV
-                                    ),
-                                )
-                                for file_id in file_ids
-                            ]
-                        )
+                        ai_message_files = [
+                            FileDescriptor(
+                                id=str(file_id),
+                                type=ChatFileType.IMAGE
+                                if custom_tool_response.response_type == "image"
+                                else ChatFileType.CSV,
+                            )
+                            for file_id in file_ids
+                        ]
                         yield FileChatDisplay(
                             file_ids=[str(file_id) for file_id in file_ids]
                         )
@@ -790,8 +737,6 @@ def stream_chat_message_objects(
                             response=custom_tool_response.tool_result,
                             tool_name=custom_tool_response.tool_name,
                         )
-                elif packet.id == SEARCH_DOC_CONTENT_ID and include_contexts:
-                    yield cast(DanswerContexts, packet.response)
 
             elif isinstance(packet, StreamStopInfo):
                 pass
@@ -831,8 +776,7 @@ def stream_chat_message_objects(
                 citations_list=answer.citations,
                 db_docs=reference_db_search_docs,
             )
-            if not answer.is_cancelled():
-                yield AllCitations(citations=answer.citations)
+            yield AllCitations(citations=answer.citations)
 
         # Saving Gen AI answer and responding with message info
         tool_name_to_tool_id: dict[str, int] = {}
@@ -901,30 +845,3 @@ def stream_chat_message(
         )
         for obj in objects:
             yield get_json_line(obj.model_dump())
-
-
-@log_function_time()
-def gather_stream_for_slack(
-    packets: ChatPacketStream,
-) -> ChatDanswerBotResponse:
-    response = ChatDanswerBotResponse()
-
-    answer = ""
-    for packet in packets:
-        if isinstance(packet, DanswerAnswerPiece) and packet.answer_piece:
-            answer += packet.answer_piece
-        elif isinstance(packet, QADocsResponse):
-            response.docs = packet
-        elif isinstance(packet, StreamingError):
-            response.error_msg = packet.error
-        elif isinstance(packet, ChatMessageDetail):
-            response.chat_message_id = packet.message_id
-        elif isinstance(packet, LLMRelevanceFilterResponse):
-            response.llm_selected_doc_indices = packet.llm_selected_doc_indices
-        elif isinstance(packet, AllCitations):
-            response.citations = packet.citations
-
-    if answer:
-        response.answer = answer
-
-    return response

backend/danswer/chat/prompt_builder/utils.py

@@ -1,62 +0,0 @@
-from langchain.schema.messages import AIMessage
-from langchain.schema.messages import BaseMessage
-from langchain.schema.messages import HumanMessage
-
-from danswer.configs.constants import MessageType
-from danswer.db.models import ChatMessage
-from danswer.file_store.models import InMemoryChatFile
-from danswer.llm.models import PreviousMessage
-from danswer.llm.utils import build_content_with_imgs
-from danswer.prompts.direct_qa_prompts import PARAMATERIZED_PROMPT
-from danswer.prompts.direct_qa_prompts import PARAMATERIZED_PROMPT_WITHOUT_CONTEXT
-
-
-def build_dummy_prompt(
-    system_prompt: str, task_prompt: str, retrieval_disabled: bool
-) -> str:
-    if retrieval_disabled:
-        return PARAMATERIZED_PROMPT_WITHOUT_CONTEXT.format(
-            user_query="<USER_QUERY>",
-            system_prompt=system_prompt,
-            task_prompt=task_prompt,
-        ).strip()
-
-    return PARAMATERIZED_PROMPT.format(
-        context_docs_str="<CONTEXT_DOCS>",
-        user_query="<USER_QUERY>",
-        system_prompt=system_prompt,
-        task_prompt=task_prompt,
-    ).strip()
-
-
-def translate_danswer_msg_to_langchain(
-    msg: ChatMessage | PreviousMessage,
-) -> BaseMessage:
-    files: list[InMemoryChatFile] = []
-
-    # If the message is a `ChatMessage`, it doesn't have the downloaded files
-    # attached. Just ignore them for now.
-    if not isinstance(msg, ChatMessage):
-        files = msg.files
-    content = build_content_with_imgs(msg.message, files, message_type=msg.message_type)
-
-    if msg.message_type == MessageType.SYSTEM:
-        raise ValueError("System messages are not currently part of history")
-    if msg.message_type == MessageType.ASSISTANT:
-        return AIMessage(content=content)
-    if msg.message_type == MessageType.USER:
-        return HumanMessage(content=content)
-
-    raise ValueError(f"New message type {msg.message_type} not handled")
-
-
-def translate_history_to_basemessages(
-    history: list[ChatMessage] | list["PreviousMessage"],
-) -> tuple[list[BaseMessage], list[int]]:
-    history_basemessages = [
-        translate_danswer_msg_to_langchain(msg)
-        for msg in history
-        if msg.token_count != 0
-    ]
-    history_token_counts = [msg.token_count for msg in history if msg.token_count != 0]
-    return history_basemessages, history_token_counts

backend/danswer/configs/app_configs.py

@@ -43,6 +43,9 @@ WEB_DOMAIN = os.environ.get("WEB_DOMAIN") or "http://localhost:3000"
 AUTH_TYPE = AuthType((os.environ.get("AUTH_TYPE") or AuthType.DISABLED.value).lower())
 DISABLE_AUTH = AUTH_TYPE == AuthType.DISABLED
 
+# Necessary for cloud integration tests
+DISABLE_VERIFICATION = os.environ.get("DISABLE_VERIFICATION", "").lower() == "true"
+
 # Encryption key secret is used to encrypt connector credentials, api keys, and other sensitive
 # information. This provides an extra layer of security on top of Postgres access controls
 # and is available in Danswer EE
@@ -81,14 +84,7 @@ OAUTH_CLIENT_SECRET = (
     or ""
 )
 
-# for future OAuth connector support
-# OAUTH_CONFLUENCE_CLIENT_ID = os.environ.get("OAUTH_CONFLUENCE_CLIENT_ID", "")
-# OAUTH_CONFLUENCE_CLIENT_SECRET = os.environ.get("OAUTH_CONFLUENCE_CLIENT_SECRET", "")
-# OAUTH_JIRA_CLIENT_ID = os.environ.get("OAUTH_JIRA_CLIENT_ID", "")
-# OAUTH_JIRA_CLIENT_SECRET = os.environ.get("OAUTH_JIRA_CLIENT_SECRET", "")
-
 USER_AUTH_SECRET = os.environ.get("USER_AUTH_SECRET", "")
-
 # for basic auth
 REQUIRE_EMAIL_VERIFICATION = (
     os.environ.get("REQUIRE_EMAIL_VERIFICATION", "").lower() == "true"
@@ -122,8 +118,6 @@ VESPA_HOST = os.environ.get("VESPA_HOST") or "localhost"
 VESPA_CONFIG_SERVER_HOST = os.environ.get("VESPA_CONFIG_SERVER_HOST") or VESPA_HOST
 VESPA_PORT = os.environ.get("VESPA_PORT") or "8081"
 VESPA_TENANT_PORT = os.environ.get("VESPA_TENANT_PORT") or "19071"
-# the number of times to try and connect to vespa on startup before giving up
-VESPA_NUM_ATTEMPTS_ON_STARTUP = int(os.environ.get("NUM_RETRIES_ON_STARTUP") or 10)
 
 VESPA_CLOUD_URL = os.environ.get("VESPA_CLOUD_URL", "")
 
@@ -314,22 +308,6 @@ CONFLUENCE_CONNECTOR_ATTACHMENT_CHAR_COUNT_THRESHOLD = int(
     os.environ.get("CONFLUENCE_CONNECTOR_ATTACHMENT_CHAR_COUNT_THRESHOLD", 200_000)
 )
 
-# Due to breakages in the confluence API, the timezone offset must be specified client side
-# to match the user's specified timezone.
-
-# The current state of affairs:
-# CQL queries are parsed in the user's timezone and cannot be specified in UTC
-# no API retrieves the user's timezone
-# All data is returned in UTC, so we can't derive the user's timezone from that
-
-# https://community.developer.atlassian.com/t/confluence-cloud-time-zone-get-via-rest-api/35954/16
-# https://jira.atlassian.com/browse/CONFCLOUD-69670
-
-# enter as a floating point offset from UTC in hours (-24 < val < 24)
-# this will be applied globally, so it probably makes sense to transition this to per
-# connector as some point.
-CONFLUENCE_TIMEZONE_OFFSET = float(os.environ.get("CONFLUENCE_TIMEZONE_OFFSET", 0.0))
-
 JIRA_CONNECTOR_LABELS_TO_SKIP = [
     ignored_tag
     for ignored_tag in os.environ.get("JIRA_CONNECTOR_LABELS_TO_SKIP", "").split(",")
@@ -348,12 +326,6 @@ GITLAB_CONNECTOR_INCLUDE_CODE_FILES = (
     os.environ.get("GITLAB_CONNECTOR_INCLUDE_CODE_FILES", "").lower() == "true"
 )
 
-# Egnyte specific configs
-EGNYTE_LOCALHOST_OVERRIDE = os.getenv("EGNYTE_LOCALHOST_OVERRIDE")
-EGNYTE_BASE_DOMAIN = os.getenv("EGNYTE_DOMAIN")
-EGNYTE_CLIENT_ID = os.getenv("EGNYTE_CLIENT_ID")
-EGNYTE_CLIENT_SECRET = os.getenv("EGNYTE_CLIENT_SECRET")
-
 DASK_JOB_CLIENT_ENABLED = (
     os.environ.get("DASK_JOB_CLIENT_ENABLED", "").lower() == "true"
 )
@@ -417,28 +389,21 @@ LARGE_CHUNK_RATIO = 4
 # We don't want the metadata to overwhelm the actual contents of the chunk
 SKIP_METADATA_IN_CHUNK = os.environ.get("SKIP_METADATA_IN_CHUNK", "").lower() == "true"
 # Timeout to wait for job's last update before killing it, in hours
-CLEANUP_INDEXING_JOBS_TIMEOUT = int(
-    os.environ.get("CLEANUP_INDEXING_JOBS_TIMEOUT") or 3
-)
+CLEANUP_INDEXING_JOBS_TIMEOUT = int(os.environ.get("CLEANUP_INDEXING_JOBS_TIMEOUT", 3))
 
 # The indexer will warn in the logs whenver a document exceeds this threshold (in bytes)
 INDEXING_SIZE_WARNING_THRESHOLD = int(
-    os.environ.get("INDEXING_SIZE_WARNING_THRESHOLD") or 100 * 1024 * 1024
+    os.environ.get("INDEXING_SIZE_WARNING_THRESHOLD", 100 * 1024 * 1024)
 )
 
 # during indexing, will log verbose memory diff stats every x batches and at the end.
 # 0 disables this behavior and is the default.
-INDEXING_TRACER_INTERVAL = int(os.environ.get("INDEXING_TRACER_INTERVAL") or 0)
+INDEXING_TRACER_INTERVAL = int(os.environ.get("INDEXING_TRACER_INTERVAL", 0))
 
 # During an indexing attempt, specifies the number of batches which are allowed to
 # exception without aborting the attempt.
-INDEXING_EXCEPTION_LIMIT = int(os.environ.get("INDEXING_EXCEPTION_LIMIT") or 0)
+INDEXING_EXCEPTION_LIMIT = int(os.environ.get("INDEXING_EXCEPTION_LIMIT", 0))
 
-# Maximum file size in a document to be indexed
-MAX_DOCUMENT_CHARS = int(os.environ.get("MAX_DOCUMENT_CHARS") or 5_000_000)
-MAX_FILE_SIZE_BYTES = int(
-    os.environ.get("MAX_FILE_SIZE_BYTES") or 2 * 1024 * 1024 * 1024
-)  # 2GB in bytes
 
 #####
 # Miscellaneous
@@ -528,6 +493,10 @@ CONTROL_PLANE_API_BASE_URL = os.environ.get(
 # JWT configuration
 JWT_ALGORITHM = "HS256"
 
+# Super Users
+SUPER_USERS = json.loads(os.environ.get("SUPER_USERS", '["pablo@danswer.ai"]'))
+SUPER_CLOUD_API_KEY = os.environ.get("SUPER_CLOUD_API_KEY", "api_key")
+
 
 #####
 # API Key Configs
@@ -541,6 +510,3 @@ API_KEY_HASH_ROUNDS = (
 
 POD_NAME = os.environ.get("POD_NAME")
 POD_NAMESPACE = os.environ.get("POD_NAMESPACE")
-
-
-DEV_MODE = os.environ.get("DEV_MODE", "").lower() == "true"

backend/danswer/configs/chat_configs.py

@@ -3,6 +3,7 @@ import os
 
 PROMPTS_YAML = "./danswer/seeding/prompts.yaml"
 PERSONAS_YAML = "./danswer/seeding/personas.yaml"
+INPUT_PROMPT_YAML = "./danswer/seeding/input_prompts.yaml"
 
 NUM_RETURNED_HITS = 50
 # Used for LLM filtering and reranking

backend/danswer/configs/constants.py

@@ -31,8 +31,6 @@ DISABLED_GEN_AI_MSG = (
     "You can still use Danswer as a search engine."
 )
 
-DEFAULT_PERSONA_ID = 0
-
 # Postgres connection constants for application_name
 POSTGRES_WEB_APP_NAME = "web"
 POSTGRES_INDEXER_APP_NAME = "indexer"
@@ -132,7 +130,6 @@ class DocumentSource(str, Enum):
     NOT_APPLICABLE = "not_applicable"
     FRESHDESK = "freshdesk"
     FIREFLIES = "fireflies"
-    EGNYTE = "egnyte"
 
 
 DocumentSourceRequiringTenantContext: list[DocumentSource] = [DocumentSource.FILE]
@@ -262,32 +259,6 @@ class DanswerCeleryPriority(int, Enum):
     LOWEST = auto()
 
 
-class DanswerCeleryTask:
-    CHECK_FOR_CONNECTOR_DELETION = "check_for_connector_deletion_task"
-    CHECK_FOR_VESPA_SYNC_TASK = "check_for_vespa_sync_task"
-    CHECK_FOR_INDEXING = "check_for_indexing"
-    CHECK_FOR_PRUNING = "check_for_pruning"
-    CHECK_FOR_DOC_PERMISSIONS_SYNC = "check_for_doc_permissions_sync"
-    CHECK_FOR_EXTERNAL_GROUP_SYNC = "check_for_external_group_sync"
-    MONITOR_VESPA_SYNC = "monitor_vespa_sync"
-    KOMBU_MESSAGE_CLEANUP_TASK = "kombu_message_cleanup_task"
-    CONNECTOR_PERMISSION_SYNC_GENERATOR_TASK = (
-        "connector_permission_sync_generator_task"
-    )
-    UPDATE_EXTERNAL_DOCUMENT_PERMISSIONS_TASK = (
-        "update_external_document_permissions_task"
-    )
-    CONNECTOR_EXTERNAL_GROUP_SYNC_GENERATOR_TASK = (
-        "connector_external_group_sync_generator_task"
-    )
-    CONNECTOR_INDEXING_PROXY_TASK = "connector_indexing_proxy_task"
-    CONNECTOR_PRUNING_GENERATOR_TASK = "connector_pruning_generator_task"
-    DOCUMENT_BY_CC_PAIR_CLEANUP_TASK = "document_by_cc_pair_cleanup_task"
-    VESPA_METADATA_SYNC_TASK = "vespa_metadata_sync_task"
-    CHECK_TTL_MANAGEMENT_TASK = "check_ttl_management_task"
-    AUTOGENERATE_USAGE_REPORT_TASK = "autogenerate_usage_report_task"
-
-
 REDIS_SOCKET_KEEPALIVE_OPTIONS = {}
 REDIS_SOCKET_KEEPALIVE_OPTIONS[socket.TCP_KEEPINTVL] = 15
 REDIS_SOCKET_KEEPALIVE_OPTIONS[socket.TCP_KEEPCNT] = 3

backend/danswer/configs/danswerbot_configs.py

@@ -4,8 +4,11 @@ import os
 # Danswer Slack Bot Configs
 #####
 DANSWER_BOT_NUM_RETRIES = int(os.environ.get("DANSWER_BOT_NUM_RETRIES", "5"))
+DANSWER_BOT_ANSWER_GENERATION_TIMEOUT = int(
+    os.environ.get("DANSWER_BOT_ANSWER_GENERATION_TIMEOUT", "90")
+)
 # How much of the available input context can be used for thread context
-MAX_THREAD_CONTEXT_PERCENTAGE = 512 * 2 / 3072
+DANSWER_BOT_TARGET_CHUNK_PERCENTAGE = 512 * 2 / 3072
 # Number of docs to display in "Reference Documents"
 DANSWER_BOT_NUM_DOCS_TO_DISPLAY = int(
     os.environ.get("DANSWER_BOT_NUM_DOCS_TO_DISPLAY", "5")
@@ -44,6 +47,17 @@ DANSWER_BOT_DISPLAY_ERROR_MSGS = os.environ.get(
 DANSWER_BOT_RESPOND_EVERY_CHANNEL = (
     os.environ.get("DANSWER_BOT_RESPOND_EVERY_CHANNEL", "").lower() == "true"
 )
+# Add a second LLM call post Answer to verify if the Answer is valid
+# Throws out answers that don't directly or fully answer the user query
+# This is the default for all DanswerBot channels unless the channel is configured individually
+# Set/unset by "Hide Non Answers"
+ENABLE_DANSWERBOT_REFLEXION = (
+    os.environ.get("ENABLE_DANSWERBOT_REFLEXION", "").lower() == "true"
+)
+# Currently not support chain of thought, probably will add back later
+DANSWER_BOT_DISABLE_COT = True
+# if set, will default DanswerBot to use quotes and reference documents
+DANSWER_BOT_USE_QUOTES = os.environ.get("DANSWER_BOT_USE_QUOTES", "").lower() == "true"
 
 # Maximum Questions Per Minute, Default Uncapped
 DANSWER_BOT_MAX_QPM = int(os.environ.get("DANSWER_BOT_MAX_QPM") or 0) or None

backend/danswer/configs/tool_configs.py

@@ -2,8 +2,6 @@ import json
 import os
 
 
-IMAGE_GENERATION_OUTPUT_FORMAT = os.environ.get("IMAGE_GENERATION_OUTPUT_FORMAT", "url")
-
 # if specified, will pass through request headers to the call to API calls made by custom tools
 CUSTOM_TOOL_PASS_THROUGH_HEADERS: list[str] | None = None
 _CUSTOM_TOOL_PASS_THROUGH_HEADERS_RAW = os.environ.get(

backend/danswer/connectors/README.md

@@ -11,16 +11,11 @@ Connectors come in 3 different flows:
 - Load Connector:
   - Bulk indexes documents to reflect a point in time. This type of connector generally works by either pulling all
   documents via a connector's API or loads the documents from some sort of a dump file.
-- Poll Connector:
+- Poll connector:
   - Incrementally updates documents based on a provided time range. It is used by the background job to pull the latest
   changes and additions since the last round of polling. This connector helps keep the document index up to date
   without needing to fetch/embed/index every document which would be too slow to do frequently on large sets of
   documents.
-- Slim Connector:
-  - This connector should be a lighter weight method of checking all documents in the source to see if they still exist.
-  - This connector should be identical to the Poll or Load Connector except that it only fetches the IDs of the documents, not the documents themselves.
-  - This is used by our pruning job which removes old documents from the index.
-  - The optional start and end datetimes can be ignored.
 - Event Based connectors:
   - Connectors that listen to events and update documents accordingly.
   - Currently not used by the background job, this exists for future design purposes.
@@ -31,14 +26,8 @@ Refer to [interfaces.py](https://github.com/danswer-ai/danswer/blob/main/backend
 and this first contributor created Pull Request for a new connector (Shoutout to Dan Brown):
 [Reference Pull Request](https://github.com/danswer-ai/danswer/pull/139)
 
-For implementing a Slim Connector, refer to the comments in this PR:
-[Slim Connector PR](https://github.com/danswer-ai/danswer/pull/3303/files)
-
-All new connectors should have tests added to the `backend/tests/daily/connectors` directory. Refer to the above PR for an example of adding tests for a new connector.
-
-
 #### Implementing the new Connector
-The connector must subclass one or more of LoadConnector, PollConnector, SlimConnector, or EventConnector.
+The connector must subclass one or more of LoadConnector, PollConnector, or EventConnector.
 
 The `__init__` should take arguments for configuring what documents the connector will and where it finds those
 documents. For example, if you have a wiki site, it may include the configuration for the team, topic, folder, etc. of

backend/danswer/connectors/confluence/connector.py

@@ -1,11 +1,9 @@
 from datetime import datetime
-from datetime import timedelta
 from datetime import timezone
 from typing import Any
 from urllib.parse import quote
 
 from danswer.configs.app_configs import CONFLUENCE_CONNECTOR_LABELS_TO_SKIP
-from danswer.configs.app_configs import CONFLUENCE_TIMEZONE_OFFSET
 from danswer.configs.app_configs import CONTINUE_ON_CONNECTOR_FAILURE
 from danswer.configs.app_configs import INDEX_BATCH_SIZE
 from danswer.configs.constants import DocumentSource
@@ -15,7 +13,6 @@ from danswer.connectors.confluence.utils import attachment_to_content
 from danswer.connectors.confluence.utils import build_confluence_document_id
 from danswer.connectors.confluence.utils import datetime_from_string
 from danswer.connectors.confluence.utils import extract_text_from_confluence_html
-from danswer.connectors.confluence.utils import validate_attachment_filetype
 from danswer.connectors.interfaces import GenerateDocumentsOutput
 from danswer.connectors.interfaces import GenerateSlimDocumentOutput
 from danswer.connectors.interfaces import LoadConnector
@@ -72,7 +69,6 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
         # skip it. This is generally used to avoid indexing extra sensitive
         # pages.
         labels_to_skip: list[str] = CONFLUENCE_CONNECTOR_LABELS_TO_SKIP,
-        timezone_offset: float = CONFLUENCE_TIMEZONE_OFFSET,
     ) -> None:
         self.batch_size = batch_size
         self.continue_on_failure = continue_on_failure
@@ -108,8 +104,6 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
             )
             self.cql_label_filter = f" and label not in ({comma_separated_labels})"
 
-        self.timezone: timezone = timezone(offset=timedelta(hours=timezone_offset))
-
     @property
     def confluence_client(self) -> OnyxConfluence:
         if self._confluence_client is None:
@@ -210,14 +204,12 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
         confluence_page_ids: list[str] = []
 
         page_query = self.cql_page_query + self.cql_label_filter + self.cql_time_filter
-        logger.debug(f"page_query: {page_query}")
         # Fetch pages as Documents
         for page in self.confluence_client.paginated_cql_retrieval(
             cql=page_query,
             expand=",".join(_PAGE_EXPANSION_FIELDS),
             limit=self.batch_size,
         ):
-            logger.debug(f"_fetch_document_batches: {page['id']}")
             confluence_page_ids.append(page["id"])
             doc = self._convert_object_to_document(page)
             if doc is not None:
@@ -250,10 +242,10 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
 
     def poll_source(self, start: float, end: float) -> GenerateDocumentsOutput:
         # Add time filters
-        formatted_start_time = datetime.fromtimestamp(start, tz=self.timezone).strftime(
+        formatted_start_time = datetime.fromtimestamp(start, tz=timezone.utc).strftime(
             "%Y-%m-%d %H:%M"
         )
-        formatted_end_time = datetime.fromtimestamp(end, tz=self.timezone).strftime(
+        formatted_end_time = datetime.fromtimestamp(end, tz=timezone.utc).strftime(
             "%Y-%m-%d %H:%M"
         )
         self.cql_time_filter = f" and lastmodified >= '{formatted_start_time}'"
@@ -277,11 +269,9 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
         ):
             # If the page has restrictions, add them to the perm_sync_data
             # These will be used by doc_sync.py to sync permissions
-            page_restrictions = page.get("restrictions")
-            page_space_key = page.get("space", {}).get("key")
-            page_perm_sync_data = {
-                "restrictions": page_restrictions or {},
-                "space_key": page_space_key,
+            perm_sync_data = {
+                "restrictions": page.get("restrictions", {}),
+                "space_key": page.get("space", {}).get("key"),
             }
 
             doc_metadata_list.append(
@@ -291,7 +281,7 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
                         page["_links"]["webui"],
                         self.is_cloud,
                     ),
-                    perm_sync_data=page_perm_sync_data,
+                    perm_sync_data=perm_sync_data,
                 )
             )
             attachment_cql = f"type=attachment and container='{page['id']}'"
@@ -301,21 +291,6 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
                 expand=restrictions_expand,
                 limit=_SLIM_DOC_BATCH_SIZE,
             ):
-                if not validate_attachment_filetype(attachment):
-                    continue
-                attachment_restrictions = attachment.get("restrictions")
-                if not attachment_restrictions:
-                    attachment_restrictions = page_restrictions
-
-                attachment_space_key = attachment.get("space", {}).get("key")
-                if not attachment_space_key:
-                    attachment_space_key = page_space_key
-
-                attachment_perm_sync_data = {
-                    "restrictions": attachment_restrictions or {},
-                    "space_key": attachment_space_key,
-                }
-
                 doc_metadata_list.append(
                     SlimDocument(
                         id=build_confluence_document_id(
@@ -323,7 +298,7 @@ class ConfluenceConnector(LoadConnector, PollConnector, SlimConnector):
                             attachment["_links"]["webui"],
                             self.is_cloud,
                         ),
-                        perm_sync_data=attachment_perm_sync_data,
+                        perm_sync_data=perm_sync_data,
                     )
                 )
             if len(doc_metadata_list) > _SLIM_DOC_BATCH_SIZE:

backend/danswer/connectors/confluence/onyx_confluence.py

@@ -134,32 +134,6 @@ class OnyxConfluence(Confluence):
         super(OnyxConfluence, self).__init__(url, *args, **kwargs)
         self._wrap_methods()
 
-    def get_current_user(self, expand: str | None = None) -> Any:
-        """
-        Implements a method that isn't in the third party client.
-
-        Get information about the current user
-        :param expand: OPTIONAL expand for get status of user.
-                Possible param is "status". Results are "Active, Deactivated"
-        :return: Returns the user details
-        """
-
-        from atlassian.errors import ApiPermissionError  # type:ignore
-
-        url = "rest/api/user/current"
-        params = {}
-        if expand:
-            params["expand"] = expand
-        try:
-            response = self.get(url, params=params)
-        except HTTPError as e:
-            if e.response.status_code == 403:
-                raise ApiPermissionError(
-                    "The calling user does not have permission", reason=e
-                )
-            raise
-        return response
-
     def _wrap_methods(self) -> None:
         """
         For each attribute that is callable (i.e., a method) and doesn't start with an underscore,
@@ -332,13 +306,6 @@ def _validate_connector_configuration(
     )
     spaces = confluence_client_with_minimal_retries.get_all_spaces(limit=1)
 
-    # uncomment the following for testing
-    # the following is an attempt to retrieve the user's timezone
-    # Unfornately, all data is returned in UTC regardless of the user's time zone
-    # even tho CQL parses incoming times based on the user's time zone
-    # space_key = spaces["results"][0]["key"]
-    # space_details = confluence_client_with_minimal_retries.cql(f"space.key={space_key}+AND+type=space")
-
     if not spaces:
         raise RuntimeError(
             f"No spaces found at {wiki_base}! "
@@ -368,5 +335,4 @@ def build_confluence_client(
         backoff_and_retry=True,
         max_backoff_retries=10,
         max_backoff_seconds=60,
-        cloud=is_cloud,
     )

backend/danswer/connectors/confluence/utils.py

@@ -32,11 +32,7 @@ def get_user_email_from_username__server(
             response = confluence_client.get_mobile_parameters(user_name)
             email = response.get("email")
         except Exception:
-            # For now, we'll just return a string that indicates failure
-            # We may want to revert to returning None in the future
-            # email = None
-            email = f"FAILED TO GET CONFLUENCE EMAIL FOR {user_name}"
-            logger.warning(f"failed to get confluence email for {user_name}")
+            email = None
         _USER_EMAIL_CACHE[user_name] = email
     return _USER_EMAIL_CACHE[user_name]
 
@@ -177,23 +173,19 @@ def extract_text_from_confluence_html(
     return format_document_soup(soup)
 
 
-def validate_attachment_filetype(attachment: dict[str, Any]) -> bool:
-    return attachment["metadata"]["mediaType"] not in [
+def attachment_to_content(
+    confluence_client: OnyxConfluence,
+    attachment: dict[str, Any],
+) -> str | None:
+    """If it returns None, assume that we should skip this attachment."""
+    if attachment["metadata"]["mediaType"] in [
         "image/jpeg",
         "image/png",
         "image/gif",
         "image/svg+xml",
         "video/mp4",
         "video/quicktime",
-    ]
-
-
-def attachment_to_content(
-    confluence_client: OnyxConfluence,
-    attachment: dict[str, Any],
-) -> str | None:
-    """If it returns None, assume that we should skip this attachment."""
-    if not validate_attachment_filetype(attachment):
+    ]:
         return None
 
     download_link = confluence_client.url + attachment["_links"]["download"]
@@ -249,7 +241,7 @@ def build_confluence_document_id(
     return f"{base_url}{content_url}"
 
 
-def _extract_referenced_attachment_names(page_text: str) -> list[str]:
+def extract_referenced_attachment_names(page_text: str) -> list[str]:
     """Parse a Confluence html page to generate a list of current
         attachments in use
 

backend/danswer/connectors/egnyte/connector.py

@@ -1,384 +0,0 @@
-import io
-import os
-from collections.abc import Generator
-from datetime import datetime
-from datetime import timezone
-from logging import Logger
-from typing import Any
-from typing import cast
-from typing import IO
-
-import requests
-from retry import retry
-
-from danswer.configs.app_configs import EGNYTE_BASE_DOMAIN
-from danswer.configs.app_configs import EGNYTE_CLIENT_ID
-from danswer.configs.app_configs import EGNYTE_CLIENT_SECRET
-from danswer.configs.app_configs import EGNYTE_LOCALHOST_OVERRIDE
-from danswer.configs.app_configs import INDEX_BATCH_SIZE
-from danswer.configs.constants import DocumentSource
-from danswer.connectors.interfaces import GenerateDocumentsOutput
-from danswer.connectors.interfaces import LoadConnector
-from danswer.connectors.interfaces import OAuthConnector
-from danswer.connectors.interfaces import PollConnector
-from danswer.connectors.interfaces import SecondsSinceUnixEpoch
-from danswer.connectors.models import BasicExpertInfo
-from danswer.connectors.models import ConnectorMissingCredentialError
-from danswer.connectors.models import Document
-from danswer.connectors.models import Section
-from danswer.file_processing.extract_file_text import detect_encoding
-from danswer.file_processing.extract_file_text import extract_file_text
-from danswer.file_processing.extract_file_text import get_file_ext
-from danswer.file_processing.extract_file_text import is_text_file_extension
-from danswer.file_processing.extract_file_text import is_valid_file_ext
-from danswer.file_processing.extract_file_text import read_text_file
-from danswer.utils.logger import setup_logger
-
-
-logger = setup_logger()
-
-_EGNYTE_API_BASE = "https://{domain}.egnyte.com/pubapi/v1"
-_EGNYTE_APP_BASE = "https://{domain}.egnyte.com"
-_TIMEOUT = 60
-
-
-def _request_with_retries(
-    method: str,
-    url: str,
-    data: dict[str, Any] | None = None,
-    headers: dict[str, Any] | None = None,
-    params: dict[str, Any] | None = None,
-    timeout: int = _TIMEOUT,
-    stream: bool = False,
-    tries: int = 8,
-    delay: float = 1,
-    backoff: float = 2,
-) -> requests.Response:
-    @retry(tries=tries, delay=delay, backoff=backoff, logger=cast(Logger, logger))
-    def _make_request() -> requests.Response:
-        response = requests.request(
-            method,
-            url,
-            data=data,
-            headers=headers,
-            params=params,
-            timeout=timeout,
-            stream=stream,
-        )
-        try:
-            response.raise_for_status()
-        except requests.exceptions.HTTPError as e:
-            if e.response.status_code != 403:
-                logger.exception(
-                    f"Failed to call Egnyte API.\n"
-                    f"URL: {url}\n"
-                    f"Headers: {headers}\n"
-                    f"Data: {data}\n"
-                    f"Params: {params}"
-                )
-            raise e
-        return response
-
-    return _make_request()
-
-
-def _parse_last_modified(last_modified: str) -> datetime:
-    return datetime.strptime(last_modified, "%a, %d %b %Y %H:%M:%S %Z").replace(
-        tzinfo=timezone.utc
-    )
-
-
-def _process_egnyte_file(
-    file_metadata: dict[str, Any],
-    file_content: IO,
-    base_url: str,
-    folder_path: str | None = None,
-) -> Document | None:
-    """Process an Egnyte file into a Document object
-
-    Args:
-        file_data: The file data from Egnyte API
-        file_content: The raw content of the file in bytes
-        base_url: The base URL for the Egnyte instance
-        folder_path: Optional folder path to filter results
-    """
-    # Skip if file path doesn't match folder path filter
-    if folder_path and not file_metadata["path"].startswith(folder_path):
-        raise ValueError(
-            f"File path {file_metadata['path']} does not match folder path {folder_path}"
-        )
-
-    file_name = file_metadata["name"]
-    extension = get_file_ext(file_name)
-    if not is_valid_file_ext(extension):
-        logger.warning(f"Skipping file '{file_name}' with extension '{extension}'")
-        return None
-
-    # Extract text content based on file type
-    if is_text_file_extension(file_name):
-        encoding = detect_encoding(file_content)
-        file_content_raw, file_metadata = read_text_file(
-            file_content, encoding=encoding, ignore_danswer_metadata=False
-        )
-    else:
-        file_content_raw = extract_file_text(
-            file=file_content,
-            file_name=file_name,
-            break_on_unprocessable=True,
-        )
-
-    # Build the web URL for the file
-    web_url = f"{base_url}/navigate/file/{file_metadata['group_id']}"
-
-    # Create document metadata
-    metadata: dict[str, str | list[str]] = {
-        "file_path": file_metadata["path"],
-        "last_modified": file_metadata.get("last_modified", ""),
-    }
-
-    # Add lock info if present
-    if lock_info := file_metadata.get("lock_info"):
-        metadata[
-            "lock_owner"
-        ] = f"{lock_info.get('first_name', '')} {lock_info.get('last_name', '')}"
-
-    # Create the document owners
-    primary_owner = None
-    if uploaded_by := file_metadata.get("uploaded_by"):
-        primary_owner = BasicExpertInfo(
-            email=uploaded_by,  # Using username as email since that's what we have
-        )
-
-    # Create the document
-    return Document(
-        id=f"egnyte-{file_metadata['entry_id']}",
-        sections=[Section(text=file_content_raw.strip(), link=web_url)],
-        source=DocumentSource.EGNYTE,
-        semantic_identifier=file_name,
-        metadata=metadata,
-        doc_updated_at=(
-            _parse_last_modified(file_metadata["last_modified"])
-            if "last_modified" in file_metadata
-            else None
-        ),
-        primary_owners=[primary_owner] if primary_owner else None,
-    )
-
-
-class EgnyteConnector(LoadConnector, PollConnector, OAuthConnector):
-    def __init__(
-        self,
-        folder_path: str | None = None,
-        batch_size: int = INDEX_BATCH_SIZE,
-    ) -> None:
-        self.domain = ""  # will always be set in `load_credentials`
-        self.folder_path = folder_path or ""  # Root folder if not specified
-        self.batch_size = batch_size
-        self.access_token: str | None = None
-
-    @classmethod
-    def oauth_id(cls) -> DocumentSource:
-        return DocumentSource.EGNYTE
-
-    @classmethod
-    def oauth_authorization_url(cls, base_domain: str, state: str) -> str:
-        if not EGNYTE_CLIENT_ID:
-            raise ValueError("EGNYTE_CLIENT_ID environment variable must be set")
-        if not EGNYTE_BASE_DOMAIN:
-            raise ValueError("EGNYTE_DOMAIN environment variable must be set")
-
-        if EGNYTE_LOCALHOST_OVERRIDE:
-            base_domain = EGNYTE_LOCALHOST_OVERRIDE
-
-        callback_uri = f"{base_domain.strip('/')}/connector/oauth/callback/egnyte"
-        return (
-            f"https://{EGNYTE_BASE_DOMAIN}.egnyte.com/puboauth/token"
-            f"?client_id={EGNYTE_CLIENT_ID}"
-            f"&redirect_uri={callback_uri}"
-            f"&scope=Egnyte.filesystem"
-            f"&state={state}"
-            f"&response_type=code"
-        )
-
-    @classmethod
-    def oauth_code_to_token(cls, code: str) -> dict[str, Any]:
-        if not EGNYTE_CLIENT_ID:
-            raise ValueError("EGNYTE_CLIENT_ID environment variable must be set")
-        if not EGNYTE_CLIENT_SECRET:
-            raise ValueError("EGNYTE_CLIENT_SECRET environment variable must be set")
-        if not EGNYTE_BASE_DOMAIN:
-            raise ValueError("EGNYTE_DOMAIN environment variable must be set")
-
-        # Exchange code for token
-        url = f"https://{EGNYTE_BASE_DOMAIN}.egnyte.com/puboauth/token"
-        data = {
-            "client_id": EGNYTE_CLIENT_ID,
-            "client_secret": EGNYTE_CLIENT_SECRET,
-            "code": code,
-            "grant_type": "authorization_code",
-            "redirect_uri": f"{EGNYTE_LOCALHOST_OVERRIDE or ''}/connector/oauth/callback/egnyte",
-            "scope": "Egnyte.filesystem",
-        }
-        headers = {"Content-Type": "application/x-www-form-urlencoded"}
-
-        response = _request_with_retries(
-            method="POST",
-            url=url,
-            data=data,
-            headers=headers,
-            # try a lot faster since this is a realtime flow
-            backoff=0,
-            delay=0.1,
-        )
-        if not response.ok:
-            raise RuntimeError(f"Failed to exchange code for token: {response.text}")
-
-        token_data = response.json()
-        return {
-            "domain": EGNYTE_BASE_DOMAIN,
-            "access_token": token_data["access_token"],
-        }
-
-    def load_credentials(self, credentials: dict[str, Any]) -> dict[str, Any] | None:
-        self.domain = credentials["domain"]
-        self.access_token = credentials["access_token"]
-        return None
-
-    def _get_files_list(
-        self,
-        path: str,
-    ) -> list[dict[str, Any]]:
-        if not self.access_token or not self.domain:
-            raise ConnectorMissingCredentialError("Egnyte")
-
-        headers = {
-            "Authorization": f"Bearer {self.access_token}",
-        }
-
-        params: dict[str, Any] = {
-            "list_content": True,
-        }
-
-        url = f"{_EGNYTE_API_BASE.format(domain=self.domain)}/fs/{path or ''}"
-        response = _request_with_retries(
-            method="GET", url=url, headers=headers, params=params, timeout=_TIMEOUT
-        )
-        if not response.ok:
-            raise RuntimeError(f"Failed to fetch files from Egnyte: {response.text}")
-
-        data = response.json()
-        all_files: list[dict[str, Any]] = []
-
-        # Add files from current directory
-        all_files.extend(data.get("files", []))
-
-        # Recursively traverse folders
-        for item in data.get("folders", []):
-            all_files.extend(self._get_files_list(item["path"]))
-
-        return all_files
-
-    def _filter_files(
-        self,
-        files: list[dict[str, Any]],
-        start_time: datetime | None = None,
-        end_time: datetime | None = None,
-    ) -> list[dict[str, Any]]:
-        filtered_files = []
-        for file in files:
-            if file["is_folder"]:
-                continue
-
-            file_modified = _parse_last_modified(file["last_modified"])
-            if start_time and file_modified < start_time:
-                continue
-            if end_time and file_modified > end_time:
-                continue
-
-            filtered_files.append(file)
-
-        return filtered_files
-
-    def _process_files(
-        self,
-        start_time: datetime | None = None,
-        end_time: datetime | None = None,
-    ) -> Generator[list[Document], None, None]:
-        files = self._get_files_list(self.folder_path)
-        files = self._filter_files(files, start_time, end_time)
-
-        current_batch: list[Document] = []
-        for file in files:
-            try:
-                # Set up request with streaming enabled
-                headers = {
-                    "Authorization": f"Bearer {self.access_token}",
-                }
-                url = f"{_EGNYTE_API_BASE.format(domain=self.domain)}/fs-content/{file['path']}"
-                response = _request_with_retries(
-                    method="GET",
-                    url=url,
-                    headers=headers,
-                    timeout=_TIMEOUT,
-                    stream=True,
-                )
-
-                if not response.ok:
-                    logger.error(
-                        f"Failed to fetch file content: {file['path']} (status code: {response.status_code})"
-                    )
-                    continue
-
-                # Stream the response content into a BytesIO buffer
-                buffer = io.BytesIO()
-                for chunk in response.iter_content(chunk_size=8192):
-                    if chunk:
-                        buffer.write(chunk)
-
-                # Reset buffer's position to the start
-                buffer.seek(0)
-
-                # Process the streamed file content
-                doc = _process_egnyte_file(
-                    file_metadata=file,
-                    file_content=buffer,
-                    base_url=_EGNYTE_APP_BASE.format(domain=self.domain),
-                    folder_path=self.folder_path,
-                )
-
-                if doc is not None:
-                    current_batch.append(doc)
-
-                    if len(current_batch) >= self.batch_size:
-                        yield current_batch
-                        current_batch = []
-
-            except Exception:
-                logger.exception(f"Failed to process file {file['path']}")
-                continue
-
-        if current_batch:
-            yield current_batch
-
-    def load_from_state(self) -> GenerateDocumentsOutput:
-        yield from self._process_files()
-
-    def poll_source(
-        self, start: SecondsSinceUnixEpoch, end: SecondsSinceUnixEpoch
-    ) -> GenerateDocumentsOutput:
-        start_time = datetime.fromtimestamp(start, tz=timezone.utc)
-        end_time = datetime.fromtimestamp(end, tz=timezone.utc)
-
-        yield from self._process_files(start_time=start_time, end_time=end_time)
-
-
-if __name__ == "__main__":
-    connector = EgnyteConnector()
-    connector.load_credentials(
-        {
-            "domain": os.environ["EGNYTE_DOMAIN"],
-            "access_token": os.environ["EGNYTE_ACCESS_TOKEN"],
-        }
-    )
-    document_batches = connector.load_from_state()
-    print(next(document_batches))

backend/danswer/connectors/factory.py

@@ -15,7 +15,6 @@ from danswer.connectors.danswer_jira.connector import JiraConnector
 from danswer.connectors.discourse.connector import DiscourseConnector
 from danswer.connectors.document360.connector import Document360Connector
 from danswer.connectors.dropbox.connector import DropboxConnector
-from danswer.connectors.egnyte.connector import EgnyteConnector
 from danswer.connectors.file.connector import LocalFileConnector
 from danswer.connectors.fireflies.connector import FirefliesConnector
 from danswer.connectors.freshdesk.connector import FreshdeskConnector
@@ -41,6 +40,7 @@ from danswer.connectors.salesforce.connector import SalesforceConnector
 from danswer.connectors.sharepoint.connector import SharepointConnector
 from danswer.connectors.slab.connector import SlabConnector
 from danswer.connectors.slack.connector import SlackPollConnector
+from danswer.connectors.slack.load_connector import SlackLoadConnector
 from danswer.connectors.teams.connector import TeamsConnector
 from danswer.connectors.web.connector import WebConnector
 from danswer.connectors.wikipedia.connector import WikipediaConnector
@@ -63,6 +63,7 @@ def identify_connector_class(
         DocumentSource.WEB: WebConnector,
         DocumentSource.FILE: LocalFileConnector,
         DocumentSource.SLACK: {
+            InputType.LOAD_STATE: SlackLoadConnector,
             InputType.POLL: SlackPollConnector,
             InputType.SLIM_RETRIEVAL: SlackPollConnector,
         },
@@ -102,7 +103,6 @@ def identify_connector_class(
         DocumentSource.XENFORO: XenforoConnector,
         DocumentSource.FRESHDESK: FreshdeskConnector,
         DocumentSource.FIREFLIES: FirefliesConnector,
-        DocumentSource.EGNYTE: EgnyteConnector,
     }
     connector_by_source = connector_map.get(source, {})
 

backend/danswer/connectors/file/connector.py

@@ -17,11 +17,11 @@ from danswer.connectors.models import BasicExpertInfo
 from danswer.connectors.models import Document
 from danswer.connectors.models import Section
 from danswer.db.engine import get_session_with_tenant
+from danswer.file_processing.extract_file_text import check_file_ext_is_valid
 from danswer.file_processing.extract_file_text import detect_encoding
 from danswer.file_processing.extract_file_text import extract_file_text
 from danswer.file_processing.extract_file_text import get_file_ext
 from danswer.file_processing.extract_file_text import is_text_file_extension
-from danswer.file_processing.extract_file_text import is_valid_file_ext
 from danswer.file_processing.extract_file_text import load_files_from_zip
 from danswer.file_processing.extract_file_text import read_pdf_file
 from danswer.file_processing.extract_file_text import read_text_file
@@ -50,7 +50,7 @@ def _read_files_and_metadata(
             file_content, ignore_dirs=True
         ):
             yield os.path.join(directory_path, file_info.filename), file, metadata
-    elif is_valid_file_ext(extension):
+    elif check_file_ext_is_valid(extension):
         yield file_name, file_content, metadata
     else:
         logger.warning(f"Skipping file '{file_name}' with extension '{extension}'")
@@ -63,7 +63,7 @@ def _process_file(
     pdf_pass: str | None = None,
 ) -> list[Document]:
     extension = get_file_ext(file_name)
-    if not is_valid_file_ext(extension):
+    if not check_file_ext_is_valid(extension):
         logger.warning(f"Skipping file '{file_name}' with extension '{extension}'")
         return []
 

backend/danswer/connectors/google_drive/connector.py

@@ -4,13 +4,11 @@ from concurrent.futures import as_completed
 from concurrent.futures import ThreadPoolExecutor
 from functools import partial
 from typing import Any
-from typing import cast
 
 from google.oauth2.credentials import Credentials as OAuthCredentials  # type: ignore
 from google.oauth2.service_account import Credentials as ServiceAccountCredentials  # type: ignore
 
 from danswer.configs.app_configs import INDEX_BATCH_SIZE
-from danswer.configs.app_configs import MAX_FILE_SIZE_BYTES
 from danswer.configs.constants import DocumentSource
 from danswer.connectors.google_drive.doc_conversion import build_slim_document
 from danswer.connectors.google_drive.doc_conversion import (
@@ -454,14 +452,12 @@ class GoogleDriveConnector(LoadConnector, PollConnector, SlimConnector):
             if isinstance(self.creds, ServiceAccountCredentials)
             else self._manage_oauth_retrieval
         )
-        drive_files = retrieval_method(
+        return retrieval_method(
             is_slim=is_slim,
             start=start,
             end=end,
         )
 
-        return drive_files
-
     def _extract_docs_from_google_drive(
         self,
         start: SecondsSinceUnixEpoch | None = None,
@@ -477,15 +473,6 @@ class GoogleDriveConnector(LoadConnector, PollConnector, SlimConnector):
         files_to_process = []
         # Gather the files into batches to be processed in parallel
         for file in self._fetch_drive_items(is_slim=False, start=start, end=end):
-            if (
-                file.get("size")
-                and int(cast(str, file.get("size"))) > MAX_FILE_SIZE_BYTES
-            ):
-                logger.warning(
-                    f"Skipping file {file.get('name', 'Unknown')} as it is too large: {file.get('size')} bytes"
-                )
-                continue
-
             files_to_process.append(file)
             if len(files_to_process) >= LARGE_BATCH_SIZE:
                 yield from _process_files_batch(

backend/danswer/connectors/google_drive/file_retrieval.py

@@ -16,7 +16,7 @@ logger = setup_logger()
 
 FILE_FIELDS = (
     "nextPageToken, files(mimeType, id, name, permissions, modifiedTime, webViewLink, "
-    "shortcutDetails, owners(emailAddress), size)"
+    "shortcutDetails, owners(emailAddress))"
 )
 SLIM_FILE_FIELDS = (
     "nextPageToken, files(mimeType, id, name, permissions(emailAddress, type), "

backend/danswer/connectors/interfaces.py

@@ -2,7 +2,6 @@ import abc
 from collections.abc import Iterator
 from typing import Any
 
-from danswer.configs.constants import DocumentSource
 from danswer.connectors.models import Document
 from danswer.connectors.models import SlimDocument
 
@@ -65,23 +64,6 @@ class SlimConnector(BaseConnector):
         raise NotImplementedError
 
 
-class OAuthConnector(BaseConnector):
-    @classmethod
-    @abc.abstractmethod
-    def oauth_id(cls) -> DocumentSource:
-        raise NotImplementedError
-
-    @classmethod
-    @abc.abstractmethod
-    def oauth_authorization_url(cls, base_domain: str, state: str) -> str:
-        raise NotImplementedError
-
-    @classmethod
-    @abc.abstractmethod
-    def oauth_code_to_token(cls, code: str) -> dict[str, Any]:
-        raise NotImplementedError
-
-
 # Event driven
 class EventConnector(BaseConnector):
     @abc.abstractmethod

backend/danswer/connectors/linear/connector.py

@@ -132,6 +132,7 @@ class LinearConnector(LoadConnector, PollConnector):
                             branchName
                             customerTicketCount
                             description
+                            descriptionData
                             comments {
                                 nodes {
                                     url
@@ -214,6 +215,5 @@ class LinearConnector(LoadConnector, PollConnector):
 if __name__ == "__main__":
     connector = LinearConnector()
     connector.load_credentials({"linear_api_key": os.environ["LINEAR_API_KEY"]})
-
     document_batches = connector.load_from_state()
     print(next(document_batches))

backend/danswer/connectors/slab/connector.py

@@ -12,15 +12,12 @@ from dateutil import parser
 from danswer.configs.app_configs import INDEX_BATCH_SIZE
 from danswer.configs.constants import DocumentSource
 from danswer.connectors.interfaces import GenerateDocumentsOutput
-from danswer.connectors.interfaces import GenerateSlimDocumentOutput
 from danswer.connectors.interfaces import LoadConnector
 from danswer.connectors.interfaces import PollConnector
 from danswer.connectors.interfaces import SecondsSinceUnixEpoch
-from danswer.connectors.interfaces import SlimConnector
 from danswer.connectors.models import ConnectorMissingCredentialError
 from danswer.connectors.models import Document
 from danswer.connectors.models import Section
-from danswer.connectors.models import SlimDocument
 from danswer.utils.logger import setup_logger
 
 
@@ -31,8 +28,6 @@ logger = setup_logger()
 SLAB_GRAPHQL_MAX_TRIES = 10
 SLAB_API_URL = "https://api.slab.com/v1/graphql"
 
-_SLIM_BATCH_SIZE = 1000
-
 
 def run_graphql_request(
     graphql_query: dict, bot_token: str, max_tries: int = SLAB_GRAPHQL_MAX_TRIES
@@ -163,26 +158,21 @@ def get_slab_url_from_title_id(base_url: str, title: str, page_id: str) -> str:
     return urljoin(urljoin(base_url, "posts/"), url_id)
 
 
-class SlabConnector(LoadConnector, PollConnector, SlimConnector):
+class SlabConnector(LoadConnector, PollConnector):
     def __init__(
         self,
         base_url: str,
         batch_size: int = INDEX_BATCH_SIZE,
+        slab_bot_token: str | None = None,
     ) -> None:
         self.base_url = base_url
         self.batch_size = batch_size
-        self._slab_bot_token: str | None = None
+        self.slab_bot_token = slab_bot_token
 
     def load_credentials(self, credentials: dict[str, Any]) -> dict[str, Any] | None:
-        self._slab_bot_token = credentials["slab_bot_token"]
+        self.slab_bot_token = credentials["slab_bot_token"]
         return None
 
-    @property
-    def slab_bot_token(self) -> str:
-        if self._slab_bot_token is None:
-            raise ConnectorMissingCredentialError("Slab")
-        return self._slab_bot_token
-
     def _iterate_posts(
         self, time_filter: Callable[[datetime], bool] | None = None
     ) -> GenerateDocumentsOutput:
@@ -237,21 +227,3 @@ class SlabConnector(LoadConnector, PollConnector, SlimConnector):
         yield from self._iterate_posts(
             time_filter=lambda t: start_time <= t <= end_time
         )
-
-    def retrieve_all_slim_documents(
-        self,
-        start: SecondsSinceUnixEpoch | None = None,
-        end: SecondsSinceUnixEpoch | None = None,
-    ) -> GenerateSlimDocumentOutput:
-        slim_doc_batch: list[SlimDocument] = []
-        for post_id in get_all_post_ids(self.slab_bot_token):
-            slim_doc_batch.append(
-                SlimDocument(
-                    id=post_id,
-                )
-            )
-            if len(slim_doc_batch) >= _SLIM_BATCH_SIZE:
-                yield slim_doc_batch
-                slim_doc_batch = []
-        if slim_doc_batch:
-            yield slim_doc_batch

backend/danswer/connectors/slack/connector.py

@@ -134,6 +134,7 @@ def get_latest_message_time(thread: ThreadType) -> datetime:
 
 
 def thread_to_doc(
+    workspace: str,
     channel: ChannelType,
     thread: ThreadType,
     slack_cleaner: SlackTextCleaner,
@@ -170,15 +171,15 @@ def thread_to_doc(
         else first_message
     )
 
-    doc_sem_id = f"{initial_sender_name} in #{channel['name']}: {snippet}".replace(
-        "\n", " "
-    )
+    doc_sem_id = f"{initial_sender_name} in #{channel['name']}: {snippet}"
 
     return Document(
         id=f"{channel_id}__{thread[0]['ts']}",
         sections=[
             Section(
-                link=get_message_link(event=m, client=client, channel_id=channel_id),
+                link=get_message_link(
+                    event=m, workspace=workspace, channel_id=channel_id
+                ),
                 text=slack_cleaner.index_clean(cast(str, m["text"])),
             )
             for m in thread
@@ -262,6 +263,7 @@ def filter_channels(
 
 def _get_all_docs(
     client: WebClient,
+    workspace: str,
     channels: list[str] | None = None,
     channel_name_regex_enabled: bool = False,
     oldest: str | None = None,
@@ -308,6 +310,7 @@ def _get_all_docs(
                 if filtered_thread:
                     channel_docs += 1
                     yield thread_to_doc(
+                        workspace=workspace,
                         channel=channel,
                         thread=filtered_thread,
                         slack_cleaner=slack_cleaner,
@@ -370,12 +373,14 @@ def _get_all_doc_ids(
 class SlackPollConnector(PollConnector, SlimConnector):
     def __init__(
         self,
+        workspace: str,
         channels: list[str] | None = None,
         # if specified, will treat the specified channel strings as
         # regexes, and will only index channels that fully match the regexes
         channel_regex_enabled: bool = False,
         batch_size: int = INDEX_BATCH_SIZE,
     ) -> None:
+        self.workspace = workspace
         self.channels = channels
         self.channel_regex_enabled = channel_regex_enabled
         self.batch_size = batch_size
@@ -409,6 +414,7 @@ class SlackPollConnector(PollConnector, SlimConnector):
         documents: list[Document] = []
         for document in _get_all_docs(
             client=self.client,
+            workspace=self.workspace,
             channels=self.channels,
             channel_name_regex_enabled=self.channel_regex_enabled,
             # NOTE: need to impute to `None` instead of using 0.0, since Slack will
@@ -432,6 +438,7 @@ if __name__ == "__main__":
 
     slack_channel = os.environ.get("SLACK_CHANNEL")
     connector = SlackPollConnector(
+        workspace=os.environ["SLACK_WORKSPACE"],
         channels=[slack_channel] if slack_channel else None,
     )
     connector.load_credentials({"slack_bot_token": os.environ["SLACK_BOT_TOKEN"]})

backend/danswer/connectors/slack/load_connector.py

@@ -0,0 +1,140 @@
+import json
+import os
+from datetime import datetime
+from datetime import timezone
+from pathlib import Path
+from typing import Any
+from typing import cast
+
+from danswer.configs.app_configs import INDEX_BATCH_SIZE
+from danswer.configs.constants import DocumentSource
+from danswer.connectors.interfaces import GenerateDocumentsOutput
+from danswer.connectors.interfaces import LoadConnector
+from danswer.connectors.models import Document
+from danswer.connectors.models import Section
+from danswer.connectors.slack.connector import filter_channels
+from danswer.connectors.slack.utils import get_message_link
+from danswer.utils.logger import setup_logger
+
+
+logger = setup_logger()
+
+
+def get_event_time(event: dict[str, Any]) -> datetime | None:
+    ts = event.get("ts")
+    if not ts:
+        return None
+    return datetime.fromtimestamp(float(ts), tz=timezone.utc)
+
+
+class SlackLoadConnector(LoadConnector):
+    # WARNING: DEPRECATED, DO NOT USE
+    def __init__(
+        self,
+        workspace: str,
+        export_path_str: str,
+        channels: list[str] | None = None,
+        # if specified, will treat the specified channel strings as
+        # regexes, and will only index channels that fully match the regexes
+        channel_regex_enabled: bool = False,
+        batch_size: int = INDEX_BATCH_SIZE,
+    ) -> None:
+        self.workspace = workspace
+        self.channels = channels
+        self.channel_regex_enabled = channel_regex_enabled
+        self.export_path_str = export_path_str
+        self.batch_size = batch_size
+
+    def load_credentials(self, credentials: dict[str, Any]) -> dict[str, Any] | None:
+        if credentials:
+            logger.warning("Unexpected credentials provided for Slack Load Connector")
+        return None
+
+    @staticmethod
+    def _process_batch_event(
+        slack_event: dict[str, Any],
+        channel: dict[str, Any],
+        matching_doc: Document | None,
+        workspace: str,
+    ) -> Document | None:
+        if (
+            slack_event["type"] == "message"
+            and slack_event.get("subtype") != "channel_join"
+        ):
+            if matching_doc:
+                return Document(
+                    id=matching_doc.id,
+                    sections=matching_doc.sections
+                    + [
+                        Section(
+                            link=get_message_link(
+                                event=slack_event,
+                                workspace=workspace,
+                                channel_id=channel["id"],
+                            ),
+                            text=slack_event["text"],
+                        )
+                    ],
+                    source=matching_doc.source,
+                    semantic_identifier=matching_doc.semantic_identifier,
+                    title="",  # slack docs don't really have a "title"
+                    doc_updated_at=get_event_time(slack_event),
+                    metadata=matching_doc.metadata,
+                )
+
+            return Document(
+                id=slack_event["ts"],
+                sections=[
+                    Section(
+                        link=get_message_link(
+                            event=slack_event,
+                            workspace=workspace,
+                            channel_id=channel["id"],
+                        ),
+                        text=slack_event["text"],
+                    )
+                ],
+                source=DocumentSource.SLACK,
+                semantic_identifier=channel["name"],
+                title="",  # slack docs don't really have a "title"
+                doc_updated_at=get_event_time(slack_event),
+                metadata={},
+            )
+
+        return None
+
+    def load_from_state(self) -> GenerateDocumentsOutput:
+        export_path = Path(self.export_path_str)
+
+        with open(export_path / "channels.json") as f:
+            all_channels = json.load(f)
+
+        filtered_channels = filter_channels(
+            all_channels, self.channels, self.channel_regex_enabled
+        )
+
+        document_batch: dict[str, Document] = {}
+        for channel_info in filtered_channels:
+            channel_dir_path = export_path / cast(str, channel_info["name"])
+            channel_file_paths = [
+                channel_dir_path / file_name
+                for file_name in os.listdir(channel_dir_path)
+            ]
+            for path in channel_file_paths:
+                with open(path) as f:
+                    events = cast(list[dict[str, Any]], json.load(f))
+                for slack_event in events:
+                    doc = self._process_batch_event(
+                        slack_event=slack_event,
+                        channel=channel_info,
+                        matching_doc=document_batch.get(
+                            slack_event.get("thread_ts", "")
+                        ),
+                        workspace=self.workspace,
+                    )
+                    if doc:
+                        document_batch[doc.id] = doc
+                        if len(document_batch) >= self.batch_size:
+                            yield list(document_batch.values())
+
+        yield list(document_batch.values())

backend/danswer/connectors/slack/utils.py

@@ -2,7 +2,6 @@ import re
 import time
 from collections.abc import Callable
 from collections.abc import Generator
-from functools import lru_cache
 from functools import wraps
 from typing import Any
 from typing import cast
@@ -22,21 +21,19 @@ basic_retry_wrapper = retry_builder()
 _SLACK_LIMIT = 900
 
 
-@lru_cache()
-def get_base_url(token: str) -> str:
-    """Retrieve and cache the base URL of the Slack workspace based on the client token."""
-    client = WebClient(token=token)
-    return client.auth_test()["url"]
-
-
 def get_message_link(
-    event: dict[str, Any], client: WebClient, channel_id: str | None = None
+    event: dict[str, Any], workspace: str, channel_id: str | None = None
 ) -> str:
-    channel_id = channel_id or event["channel"]
-    message_ts = event["ts"]
-    response = client.chat_getPermalink(channel=channel_id, message_ts=message_ts)
-    permalink = response["permalink"]
-    return permalink
+    channel_id = channel_id or cast(
+        str, event["channel"]
+    )  # channel must either be present in the event or passed in
+    message_ts = cast(str, event["ts"])
+    message_ts_without_dot = message_ts.replace(".", "")
+    thread_ts = cast(str | None, event.get("thread_ts"))
+    return (
+        f"https://{workspace}.slack.com/archives/{channel_id}/p{message_ts_without_dot}"
+        + (f"?thread_ts={thread_ts}" if thread_ts else "")
+    )
 
 
 def _make_slack_api_call_logged(

backend/danswer/connectors/teams/connector.py

@@ -33,7 +33,7 @@ def get_created_datetime(chat_message: ChatMessage) -> datetime:
 
 def _extract_channel_members(channel: Channel) -> list[BasicExpertInfo]:
     channel_members_list: list[BasicExpertInfo] = []
-    members = channel.members.get().execute_query_retry()
+    members = channel.members.get().execute_query()
     for member in members:
         channel_members_list.append(BasicExpertInfo(display_name=member.display_name))
     return channel_members_list
@@ -51,7 +51,7 @@ def _get_threads_from_channel(
         end = end.replace(tzinfo=timezone.utc)
 
     query = channel.messages.get()
-    base_messages: list[ChatMessage] = query.execute_query_retry()
+    base_messages: list[ChatMessage] = query.execute_query()
 
     threads: list[list[ChatMessage]] = []
     for base_message in base_messages:
@@ -65,7 +65,7 @@ def _get_threads_from_channel(
             continue
 
         reply_query = base_message.replies.get_all()
-        replies = reply_query.execute_query_retry()
+        replies = reply_query.execute_query()
 
         # start a list containing the base message and its replies
         thread: list[ChatMessage] = [base_message]
@@ -82,7 +82,7 @@ def _get_channels_from_teams(
     channels_list: list[Channel] = []
     for team in teams:
         query = team.channels.get()
-        channels = query.execute_query_retry()
+        channels = query.execute_query()
         channels_list.extend(channels)
 
     return channels_list
@@ -210,7 +210,7 @@ class TeamsConnector(LoadConnector, PollConnector):
 
         teams_list: list[Team] = []
 
-        teams = self.graph_client.teams.get().execute_query_retry()
+        teams = self.graph_client.teams.get().execute_query()
 
         if len(self.requested_team_list) > 0:
             adjusted_request_strings = [
@@ -234,25 +234,14 @@ class TeamsConnector(LoadConnector, PollConnector):
             raise ConnectorMissingCredentialError("Teams")
 
         teams = self._get_all_teams()
-        logger.debug(f"Found available teams: {[str(t) for t in teams]}")
-        if not teams:
-            msg = "No teams found."
-            logger.error(msg)
-            raise ValueError(msg)
 
         channels = _get_channels_from_teams(
             teams=teams,
         )
-        logger.debug(f"Found available channels: {[c.id for c in channels]}")
-        if not channels:
-            msg = "No channels found."
-            logger.error(msg)
-            raise ValueError(msg)
 
         # goes over channels, converts them into Document objects and then yields them in batches
         doc_batch: list[Document] = []
         for channel in channels:
-            logger.debug(f"Fetching threads from channel: {channel.id}")
             thread_list = _get_threads_from_channel(channel, start=start, end=end)
             for thread in thread_list:
                 converted_doc = _convert_thread_to_document(channel, thread)
@@ -270,8 +259,8 @@ class TeamsConnector(LoadConnector, PollConnector):
     def poll_source(
         self, start: SecondsSinceUnixEpoch, end: SecondsSinceUnixEpoch
     ) -> GenerateDocumentsOutput:
-        start_datetime = datetime.fromtimestamp(start, timezone.utc)
-        end_datetime = datetime.fromtimestamp(end, timezone.utc)
+        start_datetime = datetime.utcfromtimestamp(start)
+        end_datetime = datetime.utcfromtimestamp(end)
         return self._fetch_from_teams(start=start_datetime, end=end_datetime)
 
 

backend/danswer/context/search/pipeline.py

@@ -5,11 +5,7 @@ from typing import cast
 
 from sqlalchemy.orm import Session
 
-from danswer.chat.models import PromptConfig
 from danswer.chat.models import SectionRelevancePiece
-from danswer.chat.prune_and_merge import _merge_sections
-from danswer.chat.prune_and_merge import ChunkRange
-from danswer.chat.prune_and_merge import merge_chunk_intervals
 from danswer.configs.chat_configs import DISABLE_LLM_DOC_RELEVANCE
 from danswer.context.search.enums import LLMEvaluationType
 from danswer.context.search.enums import QueryFlow
@@ -31,6 +27,10 @@ from danswer.db.models import User
 from danswer.db.search_settings import get_current_search_settings
 from danswer.document_index.factory import get_default_document_index
 from danswer.document_index.interfaces import VespaChunkRequest
+from danswer.llm.answering.models import PromptConfig
+from danswer.llm.answering.prune_and_merge import _merge_sections
+from danswer.llm.answering.prune_and_merge import ChunkRange
+from danswer.llm.answering.prune_and_merge import merge_chunk_intervals
 from danswer.llm.interfaces import LLM
 from danswer.secondary_llm_flows.agentic_evaluation import evaluate_inference_section
 from danswer.utils.logger import setup_logger

backend/danswer/danswerbot/slack/blocks.py

@@ -16,7 +16,7 @@ from slack_sdk.models.blocks import SectionBlock
 from slack_sdk.models.blocks.basic_components import MarkdownTextObject
 from slack_sdk.models.blocks.block_elements import ImageElement
 
-from danswer.chat.models import ChatDanswerBotResponse
+from danswer.chat.models import DanswerQuote
 from danswer.configs.app_configs import DISABLE_GENERATIVE_AI
 from danswer.configs.app_configs import WEB_DOMAIN
 from danswer.configs.constants import DocumentSource
@@ -40,7 +40,10 @@ from danswer.danswerbot.slack.utils import translate_vespa_highlight_to_slack
 from danswer.db.chat import get_chat_session_by_message_id
 from danswer.db.engine import get_session_with_tenant
 from danswer.db.models import ChannelConfig
+from danswer.db.models import Persona
+from danswer.one_shot_answer.models import OneShotQAResponse
 from danswer.utils.text_processing import decode_escapes
+from danswer.utils.text_processing import replace_whitespaces_w_space
 
 _MAX_BLURB_LEN = 45
 
@@ -204,8 +207,7 @@ def _build_documents_blocks(
             continue
         seen_docs_identifiers.add(d.document_id)
 
-        # Strip newlines from the semantic identifier for Slackbot formatting
-        doc_sem_id = d.semantic_identifier.replace("\n", " ")
+        doc_sem_id = d.semantic_identifier
         if d.source_type == DocumentSource.SLACK.value:
             doc_sem_id = "#" + doc_sem_id
 
@@ -325,7 +327,7 @@ def _build_sources_blocks(
 
 
 def _priority_ordered_documents_blocks(
-    answer: ChatDanswerBotResponse,
+    answer: OneShotQAResponse,
 ) -> list[Block]:
     docs_response = answer.docs if answer.docs else None
     top_docs = docs_response.top_documents if docs_response else []
@@ -348,7 +350,7 @@ def _priority_ordered_documents_blocks(
 
 
 def _build_citations_blocks(
-    answer: ChatDanswerBotResponse,
+    answer: OneShotQAResponse,
 ) -> list[Block]:
     docs_response = answer.docs if answer.docs else None
     top_docs = docs_response.top_documents if docs_response else []
@@ -367,8 +369,51 @@ def _build_citations_blocks(
     return citations_block
 
 
+def _build_quotes_block(
+    quotes: list[DanswerQuote],
+) -> list[Block]:
+    quote_lines: list[str] = []
+    doc_to_quotes: dict[str, list[str]] = {}
+    doc_to_link: dict[str, str] = {}
+    doc_to_sem_id: dict[str, str] = {}
+    for q in quotes:
+        quote = q.quote
+        doc_id = q.document_id
+        doc_link = q.link
+        doc_name = q.semantic_identifier
+        if doc_link and doc_name and doc_id and quote:
+            if doc_id not in doc_to_quotes:
+                doc_to_quotes[doc_id] = [quote]
+                doc_to_link[doc_id] = doc_link
+                doc_to_sem_id[doc_id] = (
+                    doc_name
+                    if q.source_type != DocumentSource.SLACK.value
+                    else "#" + doc_name
+                )
+            else:
+                doc_to_quotes[doc_id].append(quote)
+
+    for doc_id, quote_strs in doc_to_quotes.items():
+        quotes_str_clean = [
+            replace_whitespaces_w_space(q_str).strip() for q_str in quote_strs
+        ]
+        longest_quotes = sorted(quotes_str_clean, key=len, reverse=True)[:5]
+        single_quote_str = "\n".join([f"```{q_str}```" for q_str in longest_quotes])
+        link = doc_to_link[doc_id]
+        sem_id = doc_to_sem_id[doc_id]
+        quote_lines.append(
+            f"<{link}|{sem_id}>:\n{remove_slack_text_interactions(single_quote_str)}"
+        )
+
+    if not doc_to_quotes:
+        return []
+
+    return [SectionBlock(text="*Relevant Snippets*\n" + "\n".join(quote_lines))]
+
+
 def _build_qa_response_blocks(
-    answer: ChatDanswerBotResponse,
+    answer: OneShotQAResponse,
+    skip_quotes: bool = False,
     process_message_for_citations: bool = False,
 ) -> list[Block]:
     retrieval_info = answer.docs
@@ -377,10 +422,13 @@ def _build_qa_response_blocks(
         raise RuntimeError("Failed to retrieve docs, cannot answer question.")
 
     formatted_answer = format_slack_message(answer.answer) if answer.answer else None
+    quotes = answer.quotes.quotes if answer.quotes else None
 
     if DISABLE_GENERATIVE_AI:
         return []
 
+    quotes_blocks: list[Block] = []
+
     filter_block: Block | None = None
     if (
         retrieval_info.applied_time_cutoff
@@ -423,6 +471,16 @@ def _build_qa_response_blocks(
         answer_blocks = [
             SectionBlock(text=text) for text in _split_text(answer_processed)
         ]
+        if quotes:
+            quotes_blocks = _build_quotes_block(quotes)
+
+        # if no quotes OR `_build_quotes_block()` did not give back any blocks
+        if not quotes_blocks:
+            quotes_blocks = [
+                SectionBlock(
+                    text="*Warning*: no sources were quoted for this answer, so it may be unreliable 😔"
+                )
+            ]
 
     response_blocks: list[Block] = []
 
@@ -431,6 +489,9 @@ def _build_qa_response_blocks(
 
     response_blocks.extend(answer_blocks)
 
+    if not skip_quotes:
+        response_blocks.extend(quotes_blocks)
+
     return response_blocks
 
 
@@ -506,9 +567,10 @@ def build_follow_up_resolved_blocks(
 
 
 def build_slack_response_blocks(
-    answer: ChatDanswerBotResponse,
     tenant_id: str | None,
     message_info: SlackMessageInfo,
+    answer: OneShotQAResponse,
+    persona: Persona | None,
     channel_conf: ChannelConfig | None,
     use_citations: bool,
     feedback_reminder_id: str | None,
@@ -525,6 +587,7 @@ def build_slack_response_blocks(
 
     answer_blocks = _build_qa_response_blocks(
         answer=answer,
+        skip_quotes=persona is not None or use_citations,
         process_message_for_citations=use_citations,
     )
 
@@ -554,7 +617,8 @@ def build_slack_response_blocks(
 
     citations_blocks = []
     document_blocks = []
-    if use_citations and answer.citations:
+    if use_citations:
+        # if citations are enabled, only show cited documents
         citations_blocks = _build_citations_blocks(answer)
     else:
         document_blocks = _priority_ordered_documents_blocks(answer)
@@ -573,5 +637,4 @@ def build_slack_response_blocks(
         + web_follow_up_block
         + follow_up_block
     )
-
     return all_blocks

backend/danswer/danswerbot/slack/handlers/handle_regular_answer.py

@@ -1,6 +1,7 @@
 import functools
 from collections.abc import Callable
 from typing import Any
+from typing import cast
 from typing import Optional
 from typing import TypeVar
 
@@ -8,36 +9,46 @@ from retry import retry
 from slack_sdk import WebClient
 from slack_sdk.models.blocks import SectionBlock
 
-from danswer.chat.chat_utils import prepare_chat_message_request
-from danswer.chat.models import ChatDanswerBotResponse
-from danswer.chat.process_message import gather_stream_for_slack
-from danswer.chat.process_message import stream_chat_message_objects
 from danswer.configs.app_configs import DISABLE_GENERATIVE_AI
-from danswer.configs.constants import DEFAULT_PERSONA_ID
+from danswer.configs.danswerbot_configs import DANSWER_BOT_ANSWER_GENERATION_TIMEOUT
+from danswer.configs.danswerbot_configs import DANSWER_BOT_DISABLE_COT
 from danswer.configs.danswerbot_configs import DANSWER_BOT_DISABLE_DOCS_ONLY_ANSWER
 from danswer.configs.danswerbot_configs import DANSWER_BOT_DISPLAY_ERROR_MSGS
 from danswer.configs.danswerbot_configs import DANSWER_BOT_NUM_RETRIES
+from danswer.configs.danswerbot_configs import DANSWER_BOT_TARGET_CHUNK_PERCENTAGE
+from danswer.configs.danswerbot_configs import DANSWER_BOT_USE_QUOTES
 from danswer.configs.danswerbot_configs import DANSWER_FOLLOWUP_EMOJI
 from danswer.configs.danswerbot_configs import DANSWER_REACT_EMOJI
-from danswer.configs.danswerbot_configs import MAX_THREAD_CONTEXT_PERCENTAGE
+from danswer.configs.danswerbot_configs import ENABLE_DANSWERBOT_REFLEXION
 from danswer.context.search.enums import OptionalSearchSetting
 from danswer.context.search.models import BaseFilters
+from danswer.context.search.models import RerankingDetails
 from danswer.context.search.models import RetrievalDetails
 from danswer.danswerbot.slack.blocks import build_slack_response_blocks
 from danswer.danswerbot.slack.handlers.utils import send_team_member_message
-from danswer.danswerbot.slack.handlers.utils import slackify_message_thread
 from danswer.danswerbot.slack.models import SlackMessageInfo
 from danswer.danswerbot.slack.utils import respond_in_thread
 from danswer.danswerbot.slack.utils import SlackRateLimiter
 from danswer.danswerbot.slack.utils import update_emote_react
 from danswer.db.engine import get_session_with_tenant
+from danswer.db.models import Persona
+from danswer.db.models import SlackBotResponseType
 from danswer.db.models import SlackChannelConfig
-from danswer.db.models import User
-from danswer.db.persona import get_persona_by_id
+from danswer.db.persona import fetch_persona_by_id
+from danswer.db.search_settings import get_current_search_settings
 from danswer.db.users import get_user_by_email
-from danswer.server.query_and_chat.models import CreateChatMessageRequest
+from danswer.llm.answering.prompts.citations_prompt import (
+    compute_max_document_tokens_for_persona,
+)
+from danswer.llm.factory import get_llms_for_persona
+from danswer.llm.utils import check_number_of_tokens
+from danswer.llm.utils import get_max_input_tokens
+from danswer.one_shot_answer.answer_question import get_search_answer
+from danswer.one_shot_answer.models import DirectQARequest
+from danswer.one_shot_answer.models import OneShotQAResponse
 from danswer.utils.logger import DanswerLoggingAdapter
 
+
 srl = SlackRateLimiter()
 
 RT = TypeVar("RT")  # return type
@@ -72,14 +83,16 @@ def handle_regular_answer(
     feedback_reminder_id: str | None,
     tenant_id: str | None,
     num_retries: int = DANSWER_BOT_NUM_RETRIES,
-    thread_context_percent: float = MAX_THREAD_CONTEXT_PERCENTAGE,
+    answer_generation_timeout: int = DANSWER_BOT_ANSWER_GENERATION_TIMEOUT,
+    thread_context_percent: float = DANSWER_BOT_TARGET_CHUNK_PERCENTAGE,
     should_respond_with_error_msgs: bool = DANSWER_BOT_DISPLAY_ERROR_MSGS,
     disable_docs_only_answer: bool = DANSWER_BOT_DISABLE_DOCS_ONLY_ANSWER,
+    disable_cot: bool = DANSWER_BOT_DISABLE_COT,
+    reflexion: bool = ENABLE_DANSWERBOT_REFLEXION,
 ) -> bool:
     channel_conf = slack_channel_config.channel_config if slack_channel_config else None
 
     messages = message_info.thread_messages
-
     message_ts_to_respond_to = message_info.msg_to_respond
     is_bot_msg = message_info.is_bot_msg
     user = None
@@ -89,18 +102,9 @@ def handle_regular_answer(
                 user = get_user_by_email(message_info.email, db_session)
 
     document_set_names: list[str] | None = None
-    prompt = None
-    # If no persona is specified, use the default search based persona
-    # This way slack flow always has a persona
     persona = slack_channel_config.persona if slack_channel_config else None
-    if not persona:
-        with get_session_with_tenant(tenant_id) as db_session:
-            persona = get_persona_by_id(DEFAULT_PERSONA_ID, user, db_session)
-            document_set_names = [
-                document_set.name for document_set in persona.document_sets
-            ]
-            prompt = persona.prompts[0] if persona.prompts else None
-    else:
+    prompt = None
+    if persona:
         document_set_names = [
             document_set.name for document_set in persona.document_sets
         ]
@@ -108,26 +112,6 @@ def handle_regular_answer(
 
     should_respond_even_with_no_docs = persona.num_chunks == 0 if persona else False
 
-    # TODO: Add in support for Slack to truncate messages based on max LLM context
-    # llm, _ = get_llms_for_persona(persona)
-
-    # llm_tokenizer = get_tokenizer(
-    #     model_name=llm.config.model_name,
-    #     provider_type=llm.config.model_provider,
-    # )
-
-    # # In cases of threads, split the available tokens between docs and thread context
-    # input_tokens = get_max_input_tokens(
-    #     model_name=llm.config.model_name,
-    #     model_provider=llm.config.model_provider,
-    # )
-    # max_history_tokens = int(input_tokens * thread_context_percent)
-    # combined_message = combine_message_thread(
-    #     messages, max_tokens=max_history_tokens, llm_tokenizer=llm_tokenizer
-    # )
-
-    combined_message = slackify_message_thread(messages)
-
     bypass_acl = False
     if (
         slack_channel_config
@@ -138,6 +122,13 @@ def handle_regular_answer(
         # with non-public document sets
         bypass_acl = True
 
+    # figure out if we want to use citations or quotes
+    use_citations = (
+        not DANSWER_BOT_USE_QUOTES
+        if slack_channel_config is None
+        else slack_channel_config.response_type == SlackBotResponseType.CITATIONS
+    )
+
     if not message_ts_to_respond_to and not is_bot_msg:
         # if the message is not "/danswer" command, then it should have a message ts to respond to
         raise RuntimeError(
@@ -150,23 +141,75 @@ def handle_regular_answer(
         backoff=2,
     )
     @rate_limits(client=client, channel=channel, thread_ts=message_ts_to_respond_to)
-    def _get_slack_answer(
-        new_message_request: CreateChatMessageRequest, danswer_user: User | None
-    ) -> ChatDanswerBotResponse:
+    def _get_answer(new_message_request: DirectQARequest) -> OneShotQAResponse | None:
+        max_document_tokens: int | None = None
+        max_history_tokens: int | None = None
+
         with get_session_with_tenant(tenant_id) as db_session:
-            packets = stream_chat_message_objects(
-                new_msg_req=new_message_request,
-                user=danswer_user,
+            if len(new_message_request.messages) > 1:
+                if new_message_request.persona_config:
+                    raise RuntimeError("Slack bot does not support persona config")
+                elif new_message_request.persona_id is not None:
+                    persona = cast(
+                        Persona,
+                        fetch_persona_by_id(
+                            db_session,
+                            new_message_request.persona_id,
+                            user=None,
+                            get_editable=False,
+                        ),
+                    )
+                else:
+                    raise RuntimeError(
+                        "No persona id provided, this should never happen."
+                    )
+
+                llm, _ = get_llms_for_persona(persona)
+
+                # In cases of threads, split the available tokens between docs and thread context
+                input_tokens = get_max_input_tokens(
+                    model_name=llm.config.model_name,
+                    model_provider=llm.config.model_provider,
+                )
+                max_history_tokens = int(input_tokens * thread_context_percent)
+
+                remaining_tokens = input_tokens - max_history_tokens
+
+                query_text = new_message_request.messages[0].message
+                if persona:
+                    max_document_tokens = compute_max_document_tokens_for_persona(
+                        persona=persona,
+                        actual_user_input=query_text,
+                        max_llm_token_override=remaining_tokens,
+                    )
+                else:
+                    max_document_tokens = (
+                        remaining_tokens
+                        - 512  # Needs to be more than any of the QA prompts
+                        - check_number_of_tokens(query_text)
+                    )
+
+            if DISABLE_GENERATIVE_AI:
+                return None
+
+            # This also handles creating the query event in postgres
+            answer = get_search_answer(
+                query_req=new_message_request,
+                user=user,
+                max_document_tokens=max_document_tokens,
+                max_history_tokens=max_history_tokens,
                 db_session=db_session,
+                answer_generation_timeout=answer_generation_timeout,
+                enable_reflexion=reflexion,
                 bypass_acl=bypass_acl,
+                use_citations=use_citations,
+                danswerbot_flow=True,
             )
 
-            answer = gather_stream_for_slack(packets)
-
-        if answer.error_msg:
-            raise RuntimeError(answer.error_msg)
-
-        return answer
+            if not answer.error_msg:
+                return answer
+            else:
+                raise RuntimeError(answer.error_msg)
 
     try:
         # By leaving time_cutoff and favor_recent as None, and setting enable_auto_detect_filters
@@ -196,24 +239,26 @@ def handle_regular_answer(
             enable_auto_detect_filters=auto_detect_filters,
         )
 
+        # Always apply reranking settings if it exists, this is the non-streaming flow
         with get_session_with_tenant(tenant_id) as db_session:
-            answer_request = prepare_chat_message_request(
-                message_text=combined_message,
-                user=user,
-                persona_id=persona.id,
-                # This is not used in the Slack flow, only in the answer API
-                persona_override_config=None,
-                prompt=prompt,
-                message_ts_to_respond_to=message_ts_to_respond_to,
-                retrieval_details=retrieval_details,
-                rerank_settings=None,  # Rerank customization supported in Slack flow
-                db_session=db_session,
+            saved_search_settings = get_current_search_settings(db_session)
+
+        # This includes throwing out answer via reflexion
+        answer = _get_answer(
+            DirectQARequest(
+                messages=messages,
+                multilingual_query_expansion=saved_search_settings.multilingual_expansion
+                if saved_search_settings
+                else None,
+                prompt_id=prompt.id if prompt else None,
+                persona_id=persona.id if persona is not None else 0,
+                retrieval_options=retrieval_details,
+                chain_of_thought=not disable_cot,
+                rerank_settings=RerankingDetails.from_db_model(saved_search_settings)
+                if saved_search_settings
+                else None,
             )
-
-        answer = _get_slack_answer(
-            new_message_request=answer_request, danswer_user=user
         )
-
     except Exception as e:
         logger.exception(
             f"Unable to process message - did not successfully answer "
@@ -314,7 +359,7 @@ def handle_regular_answer(
     top_docs = retrieval_info.top_documents
     if not top_docs and not should_respond_even_with_no_docs:
         logger.error(
-            f"Unable to answer question: '{combined_message}' - no documents found"
+            f"Unable to answer question: '{answer.rephrase}' - no documents found"
         )
         # Optionally, respond in thread with the error message
         # Used primarily for debugging purposes
@@ -335,18 +380,18 @@ def handle_regular_answer(
         )
         return True
 
-    only_respond_if_citations = (
+    only_respond_with_citations_or_quotes = (
         channel_conf
         and "well_answered_postfilter" in channel_conf.get("answer_filters", [])
     )
-
+    has_citations_or_quotes = bool(answer.citations or answer.quotes)
     if (
-        only_respond_if_citations
-        and not answer.citations
+        only_respond_with_citations_or_quotes
+        and not has_citations_or_quotes
         and not message_info.bypass_filters
     ):
         logger.error(
-            f"Unable to find citations to answer: '{answer.answer}' - not answering!"
+            f"Unable to find citations or quotes to answer: '{answer.rephrase}' - not answering!"
         )
         # Optionally, respond in thread with the error message
         # Used primarily for debugging purposes
@@ -364,8 +409,9 @@ def handle_regular_answer(
         tenant_id=tenant_id,
         message_info=message_info,
         answer=answer,
+        persona=persona,
         channel_conf=channel_conf,
-        use_citations=True,  # No longer supporting quotes
+        use_citations=use_citations,
         feedback_reminder_id=feedback_reminder_id,
     )
 
@@ -373,9 +419,7 @@ def handle_regular_answer(
         respond_in_thread(
             client=client,
             channel=channel,
-            receiver_ids=[message_info.sender]
-            if message_info.is_bot_msg and message_info.sender
-            else receiver_ids,
+            receiver_ids=receiver_ids,
             text="Hello! Danswer has some results for you!",
             blocks=all_blocks,
             thread_ts=message_ts_to_respond_to,

backend/danswer/danswerbot/slack/handlers/utils.py

@@ -1,33 +1,8 @@
 from slack_sdk import WebClient
 
-from danswer.chat.models import ThreadMessage
-from danswer.configs.constants import MessageType
 from danswer.danswerbot.slack.utils import respond_in_thread
 
 
-def slackify_message_thread(messages: list[ThreadMessage]) -> str:
-    # Note: this does not handle extremely long threads, every message will be included
-    # with weaker LLMs, this could cause issues with exceeeding the token limit
-    if not messages:
-        return ""
-
-    message_strs: list[str] = []
-    for message in messages:
-        if message.role == MessageType.USER:
-            message_text = (
-                f"{message.sender or 'Unknown User'} said in Slack:\n{message.message}"
-            )
-        elif message.role == MessageType.ASSISTANT:
-            message_text = f"AI said in Slack:\n{message.message}"
-        else:
-            message_text = (
-                f"{message.role.value.upper()} said in Slack:\n{message.message}"
-            )
-        message_strs.append(message_text)
-
-    return "\n\n".join(message_strs)
-
-
 def send_team_member_message(
     client: WebClient,
     channel: str,

backend/danswer/danswerbot/slack/listener.py

@@ -19,8 +19,6 @@ from slack_sdk.socket_mode.request import SocketModeRequest
 from slack_sdk.socket_mode.response import SocketModeResponse
 from sqlalchemy.orm import Session
 
-from danswer.chat.models import ThreadMessage
-from danswer.configs.app_configs import DEV_MODE
 from danswer.configs.app_configs import POD_NAME
 from danswer.configs.app_configs import POD_NAMESPACE
 from danswer.configs.constants import DanswerRedisLocks
@@ -76,6 +74,7 @@ from danswer.db.slack_bot import fetch_slack_bots
 from danswer.key_value_store.interface import KvKeyNotFoundError
 from danswer.natural_language_processing.search_nlp_models import EmbeddingModel
 from danswer.natural_language_processing.search_nlp_models import warm_up_bi_encoder
+from danswer.one_shot_answer.models import ThreadMessage
 from danswer.redis.redis_pool import get_redis_client
 from danswer.server.manage.models import SlackBotTokens
 from danswer.utils.logger import setup_logger
@@ -251,7 +250,7 @@ class SlackbotHandler:
                 nx=True,
                 ex=TENANT_LOCK_EXPIRATION,
             )
-            if not acquired and not DEV_MODE:
+            if not acquired:
                 logger.debug(f"Another pod holds the lock for tenant {tenant_id}")
                 continue
 

backend/danswer/danswerbot/slack/models.py

@@ -1,6 +1,6 @@
 from pydantic import BaseModel
 
-from danswer.chat.models import ThreadMessage
+from danswer.one_shot_answer.models import ThreadMessage
 
 
 class SlackMessageInfo(BaseModel):

backend/danswer/danswerbot/slack/utils.py

@@ -11,7 +11,6 @@ from retry import retry
 from slack_sdk import WebClient
 from slack_sdk.errors import SlackApiError
 from slack_sdk.models.blocks import Block
-from slack_sdk.models.blocks import SectionBlock
 from slack_sdk.models.metadata import Metadata
 from slack_sdk.socket_mode import SocketModeClient
 
@@ -31,13 +30,13 @@ from danswer.configs.danswerbot_configs import (
 from danswer.connectors.slack.utils import make_slack_api_rate_limited
 from danswer.connectors.slack.utils import SlackTextCleaner
 from danswer.danswerbot.slack.constants import FeedbackVisibility
-from danswer.danswerbot.slack.models import ThreadMessage
 from danswer.db.engine import get_session_with_tenant
 from danswer.db.users import get_user_by_email
 from danswer.llm.exceptions import GenAIDisabledException
 from danswer.llm.factory import get_default_llms
 from danswer.llm.utils import dict_based_prompt_to_langchain_prompt
 from danswer.llm.utils import message_to_string
+from danswer.one_shot_answer.models import ThreadMessage
 from danswer.prompts.miscellaneous_prompts import SLACK_LANGUAGE_REPHRASE_PROMPT
 from danswer.utils.logger import setup_logger
 from danswer.utils.telemetry import optional_telemetry
@@ -141,40 +140,6 @@ def remove_danswer_bot_tag(message_str: str, client: WebClient) -> str:
     return re.sub(rf"<@{bot_tag_id}>\s", "", message_str)
 
 
-def _check_for_url_in_block(block: Block) -> bool:
-    """
-    Check if the block has a key that contains "url" in it
-    """
-    block_dict = block.to_dict()
-
-    def check_dict_for_url(d: dict) -> bool:
-        for key, value in d.items():
-            if "url" in key.lower():
-                return True
-            if isinstance(value, dict):
-                if check_dict_for_url(value):
-                    return True
-            elif isinstance(value, list):
-                for item in value:
-                    if isinstance(item, dict) and check_dict_for_url(item):
-                        return True
-        return False
-
-    return check_dict_for_url(block_dict)
-
-
-def _build_error_block(error_message: str) -> Block:
-    """
-    Build an error block to display in slack so that the user can see
-    the error without completely breaking
-    """
-    display_text = (
-        "There was an error displaying all of the Onyx answers."
-        f" Please let an admin or an onyx developer know. Error: {error_message}"
-    )
-    return SectionBlock(text=display_text)
-
-
 @retry(
     tries=DANSWER_BOT_NUM_RETRIES,
     delay=0.25,
@@ -197,9 +162,24 @@ def respond_in_thread(
     message_ids: list[str] = []
     if not receiver_ids:
         slack_call = make_slack_api_rate_limited(client.chat_postMessage)
-        try:
+        response = slack_call(
+            channel=channel,
+            text=text,
+            blocks=blocks,
+            thread_ts=thread_ts,
+            metadata=metadata,
+            unfurl_links=unfurl,
+            unfurl_media=unfurl,
+        )
+        if not response.get("ok"):
+            raise RuntimeError(f"Failed to post message: {response}")
+        message_ids.append(response["message_ts"])
+    else:
+        slack_call = make_slack_api_rate_limited(client.chat_postEphemeral)
+        for receiver in receiver_ids:
             response = slack_call(
                 channel=channel,
+                user=receiver,
                 text=text,
                 blocks=blocks,
                 thread_ts=thread_ts,
@@ -207,68 +187,8 @@ def respond_in_thread(
                 unfurl_links=unfurl,
                 unfurl_media=unfurl,
             )
-        except Exception as e:
-            logger.warning(f"Failed to post message: {e} \n blocks: {blocks}")
-            logger.warning("Trying again without blocks that have urls")
-
-            if not blocks:
-                raise e
-
-            blocks_without_urls = [
-                block for block in blocks if not _check_for_url_in_block(block)
-            ]
-            blocks_without_urls.append(_build_error_block(str(e)))
-
-            # Try again wtihout blocks containing url
-            response = slack_call(
-                channel=channel,
-                text=text,
-                blocks=blocks_without_urls,
-                thread_ts=thread_ts,
-                metadata=metadata,
-                unfurl_links=unfurl,
-                unfurl_media=unfurl,
-            )
-
-        message_ids.append(response["message_ts"])
-    else:
-        slack_call = make_slack_api_rate_limited(client.chat_postEphemeral)
-        for receiver in receiver_ids:
-            try:
-                response = slack_call(
-                    channel=channel,
-                    user=receiver,
-                    text=text,
-                    blocks=blocks,
-                    thread_ts=thread_ts,
-                    metadata=metadata,
-                    unfurl_links=unfurl,
-                    unfurl_media=unfurl,
-                )
-            except Exception as e:
-                logger.warning(f"Failed to post message: {e} \n blocks: {blocks}")
-                logger.warning("Trying again without blocks that have urls")
-
-                if not blocks:
-                    raise e
-
-                blocks_without_urls = [
-                    block for block in blocks if not _check_for_url_in_block(block)
-                ]
-                blocks_without_urls.append(_build_error_block(str(e)))
-
-                # Try again wtihout blocks containing url
-                response = slack_call(
-                    channel=channel,
-                    user=receiver,
-                    text=text,
-                    blocks=blocks_without_urls,
-                    thread_ts=thread_ts,
-                    metadata=metadata,
-                    unfurl_links=unfurl,
-                    unfurl_media=unfurl,
-                )
-
+            if not response.get("ok"):
+                raise RuntimeError(f"Failed to post message: {response}")
             message_ids.append(response["message_ts"])
 
     return message_ids

backend/danswer/db/chat.py

@@ -145,10 +145,16 @@ def get_chat_sessions_by_user(
     user_id: UUID | None,
     deleted: bool | None,
     db_session: Session,
+    only_one_shot: bool = False,
     limit: int = 50,
 ) -> list[ChatSession]:
     stmt = select(ChatSession).where(ChatSession.user_id == user_id)
 
+    if only_one_shot:
+        stmt = stmt.where(ChatSession.one_shot.is_(True))
+    else:
+        stmt = stmt.where(ChatSession.one_shot.is_(False))
+
     stmt = stmt.order_by(desc(ChatSession.time_created))
 
     if deleted is not None:
@@ -220,11 +226,12 @@ def delete_messages_and_files_from_chat_session(
 
 def create_chat_session(
     db_session: Session,
-    description: str | None,
+    description: str,
     user_id: UUID | None,
     persona_id: int | None,  # Can be none if temporary persona is used
     llm_override: LLMOverride | None = None,
     prompt_override: PromptOverride | None = None,
+    one_shot: bool = False,
     danswerbot_flow: bool = False,
     slack_thread_id: str | None = None,
 ) -> ChatSession:
@@ -234,6 +241,7 @@ def create_chat_session(
         description=description,
         llm_override=llm_override,
         prompt_override=prompt_override,
+        one_shot=one_shot,
         danswerbot_flow=danswerbot_flow,
         slack_thread_id=slack_thread_id,
     )
@@ -279,6 +287,8 @@ def duplicate_chat_session_for_user_from_slack(
         description="",
         llm_override=chat_session.llm_override,
         prompt_override=chat_session.prompt_override,
+        # Chat sessions from Slack should put people in the chat UI, not the search
+        one_shot=False,
         # Chat is in UI now so this is false
         danswerbot_flow=False,
         # Maybe we want this in the future to track if it was created from Slack

backend/danswer/db/credentials.py

@@ -20,6 +20,7 @@ from danswer.db.models import DocumentByConnectorCredentialPair
 from danswer.db.models import User
 from danswer.db.models import User__UserGroup
 from danswer.server.documents.models import CredentialBase
+from danswer.server.documents.models import CredentialDataUpdateRequest
 from danswer.utils.logger import setup_logger
 
 
@@ -247,6 +248,7 @@ def create_credential(
     )
 
     db_session.commit()
+
     return credential
 
 
@@ -261,8 +263,7 @@ def _cleanup_credential__user_group_relationships__no_commit(
 
 def alter_credential(
     credential_id: int,
-    name: str,
-    credential_json: dict[str, Any],
+    credential_data: CredentialDataUpdateRequest,
     user: User,
     db_session: Session,
 ) -> Credential | None:
@@ -272,13 +273,11 @@ def alter_credential(
     if credential is None:
         return None
 
-    credential.name = name
+    credential.name = credential_data.name
 
-    # Assign a new dictionary to credential.credential_json
-    credential.credential_json = {
-        **credential.credential_json,
-        **credential_json,
-    }
+    # Update only the keys present in credential_data.credential_json
+    for key, value in credential_data.credential_json.items():
+        credential.credential_json[key] = value
 
     credential.user_id = user.id if user is not None else None
     db_session.commit()
@@ -311,8 +310,8 @@ def update_credential_json(
     credential = fetch_credential_by_id(credential_id, user, db_session)
     if credential is None:
         return None
-
     credential.credential_json = credential_json
+
     db_session.commit()
     return credential
 

backend/danswer/db/engine.py

@@ -37,7 +37,6 @@ from danswer.configs.app_configs import POSTGRES_PORT
 from danswer.configs.app_configs import POSTGRES_USER
 from danswer.configs.app_configs import USER_AUTH_SECRET
 from danswer.configs.constants import POSTGRES_UNKNOWN_APP_NAME
-from danswer.server.utils import BasicAuthenticationError
 from danswer.utils.logger import setup_logger
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.configs import POSTGRES_DEFAULT_SCHEMA
@@ -427,9 +426,7 @@ def get_session() -> Generator[Session, None, None]:
     """Generate a database session with the appropriate tenant schema set."""
     tenant_id = CURRENT_TENANT_ID_CONTEXTVAR.get()
     if tenant_id == POSTGRES_DEFAULT_SCHEMA and MULTI_TENANT:
-        raise BasicAuthenticationError(
-            detail="User must authenticate",
-        )
+        raise HTTPException(status_code=401, detail="User must authenticate")
 
     engine = get_sqlalchemy_engine()
 

backend/danswer/db/index_attempt.py

@@ -522,16 +522,12 @@ def expire_index_attempts(
     search_settings_id: int,
     db_session: Session,
 ) -> None:
-    not_started_query = (
-        update(IndexAttempt)
+    delete_query = (
+        delete(IndexAttempt)
         .where(IndexAttempt.search_settings_id == search_settings_id)
         .where(IndexAttempt.status == IndexingStatus.NOT_STARTED)
-        .values(
-            status=IndexingStatus.CANCELED,
-            error_msg="Canceled, likely due to model swap",
-        )
     )
-    db_session.execute(not_started_query)
+    db_session.execute(delete_query)
 
     update_query = (
         update(IndexAttempt)
@@ -553,14 +549,9 @@ def cancel_indexing_attempts_for_ccpair(
     include_secondary_index: bool = False,
 ) -> None:
     stmt = (
-        update(IndexAttempt)
+        delete(IndexAttempt)
         .where(IndexAttempt.connector_credential_pair_id == cc_pair_id)
         .where(IndexAttempt.status == IndexingStatus.NOT_STARTED)
-        .values(
-            status=IndexingStatus.CANCELED,
-            error_msg="Canceled by user",
-            time_started=datetime.now(timezone.utc),
-        )
     )
 
     if not include_secondary_index:

backend/danswer/db/input_prompt.py

@@ -0,0 +1,202 @@
+from uuid import UUID
+
+from fastapi import HTTPException
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from danswer.db.models import InputPrompt
+from danswer.db.models import User
+from danswer.server.features.input_prompt.models import InputPromptSnapshot
+from danswer.server.manage.models import UserInfo
+from danswer.utils.logger import setup_logger
+
+
+logger = setup_logger()
+
+
+def insert_input_prompt_if_not_exists(
+    user: User | None,
+    input_prompt_id: int | None,
+    prompt: str,
+    content: str,
+    active: bool,
+    is_public: bool,
+    db_session: Session,
+    commit: bool = True,
+) -> InputPrompt:
+    if input_prompt_id is not None:
+        input_prompt = (
+            db_session.query(InputPrompt).filter_by(id=input_prompt_id).first()
+        )
+    else:
+        query = db_session.query(InputPrompt).filter(InputPrompt.prompt == prompt)
+        if user:
+            query = query.filter(InputPrompt.user_id == user.id)
+        else:
+            query = query.filter(InputPrompt.user_id.is_(None))
+        input_prompt = query.first()
+
+    if input_prompt is None:
+        input_prompt = InputPrompt(
+            id=input_prompt_id,
+            prompt=prompt,
+            content=content,
+            active=active,
+            is_public=is_public or user is None,
+            user_id=user.id if user else None,
+        )
+        db_session.add(input_prompt)
+
+    if commit:
+        db_session.commit()
+
+    return input_prompt
+
+
+def insert_input_prompt(
+    prompt: str,
+    content: str,
+    is_public: bool,
+    user: User | None,
+    db_session: Session,
+) -> InputPrompt:
+    input_prompt = InputPrompt(
+        prompt=prompt,
+        content=content,
+        active=True,
+        is_public=is_public or user is None,
+        user_id=user.id if user is not None else None,
+    )
+    db_session.add(input_prompt)
+    db_session.commit()
+
+    return input_prompt
+
+
+def update_input_prompt(
+    user: User | None,
+    input_prompt_id: int,
+    prompt: str,
+    content: str,
+    active: bool,
+    db_session: Session,
+) -> InputPrompt:
+    input_prompt = db_session.scalar(
+        select(InputPrompt).where(InputPrompt.id == input_prompt_id)
+    )
+    if input_prompt is None:
+        raise ValueError(f"No input prompt with id {input_prompt_id}")
+
+    if not validate_user_prompt_authorization(user, input_prompt):
+        raise HTTPException(status_code=401, detail="You don't own this prompt")
+
+    input_prompt.prompt = prompt
+    input_prompt.content = content
+    input_prompt.active = active
+
+    db_session.commit()
+    return input_prompt
+
+
+def validate_user_prompt_authorization(
+    user: User | None, input_prompt: InputPrompt
+) -> bool:
+    prompt = InputPromptSnapshot.from_model(input_prompt=input_prompt)
+
+    if prompt.user_id is not None:
+        if user is None:
+            return False
+
+        user_details = UserInfo.from_model(user)
+        if str(user_details.id) != str(prompt.user_id):
+            return False
+    return True
+
+
+def remove_public_input_prompt(input_prompt_id: int, db_session: Session) -> None:
+    input_prompt = db_session.scalar(
+        select(InputPrompt).where(InputPrompt.id == input_prompt_id)
+    )
+
+    if input_prompt is None:
+        raise ValueError(f"No input prompt with id {input_prompt_id}")
+
+    if not input_prompt.is_public:
+        raise HTTPException(status_code=400, detail="This prompt is not public")
+
+    db_session.delete(input_prompt)
+    db_session.commit()
+
+
+def remove_input_prompt(
+    user: User | None, input_prompt_id: int, db_session: Session
+) -> None:
+    input_prompt = db_session.scalar(
+        select(InputPrompt).where(InputPrompt.id == input_prompt_id)
+    )
+    if input_prompt is None:
+        raise ValueError(f"No input prompt with id {input_prompt_id}")
+
+    if input_prompt.is_public:
+        raise HTTPException(
+            status_code=400, detail="Cannot delete public prompts with this method"
+        )
+
+    if not validate_user_prompt_authorization(user, input_prompt):
+        raise HTTPException(status_code=401, detail="You do not own this prompt")
+
+    db_session.delete(input_prompt)
+    db_session.commit()
+
+
+def fetch_input_prompt_by_id(
+    id: int, user_id: UUID | None, db_session: Session
+) -> InputPrompt:
+    query = select(InputPrompt).where(InputPrompt.id == id)
+
+    if user_id:
+        query = query.where(
+            (InputPrompt.user_id == user_id) | (InputPrompt.user_id is None)
+        )
+    else:
+        # If no user_id is provided, only fetch prompts without a user_id (aka public)
+        query = query.where(InputPrompt.user_id == None)  # noqa
+
+    result = db_session.scalar(query)
+
+    if result is None:
+        raise HTTPException(422, "No input prompt found")
+
+    return result
+
+
+def fetch_public_input_prompts(
+    db_session: Session,
+) -> list[InputPrompt]:
+    query = select(InputPrompt).where(InputPrompt.is_public)
+    return list(db_session.scalars(query).all())
+
+
+def fetch_input_prompts_by_user(
+    db_session: Session,
+    user_id: UUID | None,
+    active: bool | None = None,
+    include_public: bool = False,
+) -> list[InputPrompt]:
+    query = select(InputPrompt)
+
+    if user_id is not None:
+        if include_public:
+            query = query.where(
+                (InputPrompt.user_id == user_id) | InputPrompt.is_public
+            )
+        else:
+            query = query.where(InputPrompt.user_id == user_id)
+
+    elif include_public:
+        query = query.where(InputPrompt.is_public)
+
+    if active is not None:
+        query = query.where(InputPrompt.active == active)
+
+    return list(db_session.scalars(query).all())

backend/danswer/db/models.py

@@ -1,5 +1,6 @@
 import datetime
 import json
+from enum import Enum as PyEnum
 from typing import Any
 from typing import Literal
 from typing import NotRequired
@@ -159,6 +160,9 @@ class User(SQLAlchemyBaseUserTableUUID, Base):
     )
 
     prompts: Mapped[list["Prompt"]] = relationship("Prompt", back_populates="user")
+    input_prompts: Mapped[list["InputPrompt"]] = relationship(
+        "InputPrompt", back_populates="user"
+    )
 
     # Personas owned by this user
     personas: Mapped[list["Persona"]] = relationship("Persona", back_populates="user")
@@ -175,6 +179,31 @@ class User(SQLAlchemyBaseUserTableUUID, Base):
     )
 
 
+class InputPrompt(Base):
+    __tablename__ = "inputprompt"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
+    prompt: Mapped[str] = mapped_column(String)
+    content: Mapped[str] = mapped_column(String)
+    active: Mapped[bool] = mapped_column(Boolean)
+    user: Mapped[User | None] = relationship("User", back_populates="input_prompts")
+    is_public: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
+    user_id: Mapped[UUID | None] = mapped_column(
+        ForeignKey("user.id", ondelete="CASCADE"), nullable=True
+    )
+
+
+class InputPrompt__User(Base):
+    __tablename__ = "inputprompt__user"
+
+    input_prompt_id: Mapped[int] = mapped_column(
+        ForeignKey("inputprompt.id"), primary_key=True
+    )
+    user_id: Mapped[UUID | None] = mapped_column(
+        ForeignKey("inputprompt.id"), primary_key=True
+    )
+
+
 class AccessToken(SQLAlchemyBaseAccessTokenTableUUID, Base):
     pass
 
@@ -568,25 +597,6 @@ class Connector(Base):
         list["DocumentByConnectorCredentialPair"]
     ] = relationship("DocumentByConnectorCredentialPair", back_populates="connector")
 
-    # synchronize this validation logic with RefreshFrequencySchema etc on front end
-    # until we have a centralized validation schema
-
-    # TODO(rkuo): experiment with SQLAlchemy validators rather than manual checks
-    # https://docs.sqlalchemy.org/en/20/orm/mapped_attributes.html
-    def validate_refresh_freq(self) -> None:
-        if self.refresh_freq is not None:
-            if self.refresh_freq < 60:
-                raise ValueError(
-                    "refresh_freq must be greater than or equal to 60 seconds."
-                )
-
-    def validate_prune_freq(self) -> None:
-        if self.prune_freq is not None:
-            if self.prune_freq < 86400:
-                raise ValueError(
-                    "prune_freq must be greater than or equal to 86400 seconds."
-                )
-
 
 class Credential(Base):
     __tablename__ = "credential"
@@ -954,8 +964,9 @@ class ChatSession(Base):
     persona_id: Mapped[int | None] = mapped_column(
         ForeignKey("persona.id"), nullable=True
     )
-    description: Mapped[str | None] = mapped_column(Text, nullable=True)
-    # This chat created by DanswerBot
+    description: Mapped[str] = mapped_column(Text)
+    # One-shot direct answering, currently the two types of chats are not mixed
+    one_shot: Mapped[bool] = mapped_column(Boolean, default=False)
     danswerbot_flow: Mapped[bool] = mapped_column(Boolean, default=False)
     # Only ever set to True if system is set to not hard-delete chats
     deleted: Mapped[bool] = mapped_column(Boolean, default=False)
@@ -1477,13 +1488,16 @@ class ChannelConfig(TypedDict):
     show_continue_in_web_ui: NotRequired[bool]  # defaults to False
 
 
+class SlackBotResponseType(str, PyEnum):
+    QUOTES = "quotes"
+    CITATIONS = "citations"
+
+
 class SlackChannelConfig(Base):
     __tablename__ = "slack_channel_config"
 
     id: Mapped[int] = mapped_column(primary_key=True)
-    slack_bot_id: Mapped[int] = mapped_column(
-        ForeignKey("slack_bot.id"), nullable=False
-    )
+    slack_bot_id: Mapped[int] = mapped_column(ForeignKey("slack_bot.id"), nullable=True)
     persona_id: Mapped[int | None] = mapped_column(
         ForeignKey("persona.id"), nullable=True
     )
@@ -1491,6 +1505,9 @@ class SlackChannelConfig(Base):
     channel_config: Mapped[ChannelConfig] = mapped_column(
         postgresql.JSONB(), nullable=False
     )
+    response_type: Mapped[SlackBotResponseType] = mapped_column(
+        Enum(SlackBotResponseType, native_enum=False), nullable=False
+    )
 
     enable_auto_filters: Mapped[bool] = mapped_column(
         Boolean, nullable=False, default=False
@@ -1521,7 +1538,6 @@ class SlackBot(Base):
     slack_channel_configs: Mapped[list[SlackChannelConfig]] = relationship(
         "SlackChannelConfig",
         back_populates="slack_bot",
-        cascade="all, delete-orphan",
     )
 
 

backend/danswer/db/persona.py

@@ -415,6 +415,9 @@ def upsert_prompt(
     return prompt
 
 
+# NOTE: This operation cannot update persona configuration options that
+# are core to the persona, such as its display priority and
+# whether or not the assistant is a built-in / default assistant
 def upsert_persona(
     user: User | None,
     name: str,
@@ -446,16 +449,10 @@ def upsert_persona(
     chunks_above: int = CONTEXT_CHUNKS_ABOVE,
     chunks_below: int = CONTEXT_CHUNKS_BELOW,
 ) -> Persona:
-    """
-    NOTE: This operation cannot update persona configuration options that
-    are core to the persona, such as its display priority and
-    whether or not the assistant is a built-in / default assistant
-    """
-
     if persona_id is not None:
-        existing_persona = db_session.query(Persona).filter_by(id=persona_id).first()
+        persona = db_session.query(Persona).filter_by(id=persona_id).first()
     else:
-        existing_persona = _get_persona_by_name(
+        persona = _get_persona_by_name(
             persona_name=name, user=user, db_session=db_session
         )
 
@@ -481,78 +478,57 @@ def upsert_persona(
     prompts = None
     if prompt_ids is not None:
         prompts = db_session.query(Prompt).filter(Prompt.id.in_(prompt_ids)).all()
-
-    if prompts is not None and len(prompts) == 0:
-        raise ValueError(
-            f"Invalid Persona config, no valid prompts "
-            f"specified. Specified IDs were: '{prompt_ids}'"
-        )
+        if not prompts and prompt_ids:
+            raise ValueError("prompts not found")
 
     # ensure all specified tools are valid
     if tools:
         validate_persona_tools(tools)
 
-    if existing_persona:
-        # Built-in personas can only be updated through YAML configuration.
-        # This ensures that core system personas are not modified unintentionally.
-        if existing_persona.builtin_persona and not builtin_persona:
+    if persona:
+        if persona.builtin_persona and not builtin_persona:
             raise ValueError("Cannot update builtin persona with non-builtin.")
 
         # this checks if the user has permission to edit the persona
-        # will raise an Exception if the user does not have permission
-        existing_persona = fetch_persona_by_id(
-            db_session=db_session,
-            persona_id=existing_persona.id,
-            user=user,
-            get_editable=True,
+        persona = fetch_persona_by_id(
+            db_session=db_session, persona_id=persona.id, user=user, get_editable=True
         )
 
-        # The following update excludes `default`, `built-in`, and display priority.
-        # Display priority is handled separately in the `display-priority` endpoint.
-        # `default` and `built-in` properties can only be set when creating a persona.
-        existing_persona.name = name
-        existing_persona.description = description
-        existing_persona.num_chunks = num_chunks
-        existing_persona.chunks_above = chunks_above
-        existing_persona.chunks_below = chunks_below
-        existing_persona.llm_relevance_filter = llm_relevance_filter
-        existing_persona.llm_filter_extraction = llm_filter_extraction
-        existing_persona.recency_bias = recency_bias
-        existing_persona.llm_model_provider_override = llm_model_provider_override
-        existing_persona.llm_model_version_override = llm_model_version_override
-        existing_persona.starter_messages = starter_messages
-        existing_persona.deleted = False  # Un-delete if previously deleted
-        existing_persona.is_public = is_public
-        existing_persona.icon_color = icon_color
-        existing_persona.icon_shape = icon_shape
+        persona.name = name
+        persona.description = description
+        persona.num_chunks = num_chunks
+        persona.chunks_above = chunks_above
+        persona.chunks_below = chunks_below
+        persona.llm_relevance_filter = llm_relevance_filter
+        persona.llm_filter_extraction = llm_filter_extraction
+        persona.recency_bias = recency_bias
+        persona.llm_model_provider_override = llm_model_provider_override
+        persona.llm_model_version_override = llm_model_version_override
+        persona.starter_messages = starter_messages
+        persona.deleted = False  # Un-delete if previously deleted
+        persona.is_public = is_public
+        persona.icon_color = icon_color
+        persona.icon_shape = icon_shape
         if remove_image or uploaded_image_id:
-            existing_persona.uploaded_image_id = uploaded_image_id
-        existing_persona.is_visible = is_visible
-        existing_persona.search_start_date = search_start_date
-        existing_persona.category_id = category_id
+            persona.uploaded_image_id = uploaded_image_id
+        persona.is_visible = is_visible
+        persona.search_start_date = search_start_date
+        persona.category_id = category_id
         # Do not delete any associations manually added unless
         # a new updated list is provided
         if document_sets is not None:
-            existing_persona.document_sets.clear()
-            existing_persona.document_sets = document_sets or []
+            persona.document_sets.clear()
+            persona.document_sets = document_sets or []
 
         if prompts is not None:
-            existing_persona.prompts.clear()
-            existing_persona.prompts = prompts
+            persona.prompts.clear()
+            persona.prompts = prompts or []
 
         if tools is not None:
-            existing_persona.tools = tools or []
-
-        persona = existing_persona
+            persona.tools = tools or []
 
     else:
-        if not prompts:
-            raise ValueError(
-                "Invalid Persona config. "
-                "Must specify at least one prompt for a new persona."
-            )
-
-        new_persona = Persona(
+        persona = Persona(
             id=persona_id,
             user_id=user.id if user else None,
             is_public=is_public,
@@ -565,7 +541,7 @@ def upsert_persona(
             llm_filter_extraction=llm_filter_extraction,
             recency_bias=recency_bias,
             builtin_persona=builtin_persona,
-            prompts=prompts,
+            prompts=prompts or [],
             document_sets=document_sets or [],
             llm_model_provider_override=llm_model_provider_override,
             llm_model_version_override=llm_model_version_override,
@@ -580,8 +556,8 @@ def upsert_persona(
             is_default_persona=is_default_persona,
             category_id=category_id,
         )
-        db_session.add(new_persona)
-        persona = new_persona
+        db_session.add(persona)
+
     if commit:
         db_session.commit()
     else:

backend/danswer/db/slack_channel_config.py

@@ -10,6 +10,7 @@ from danswer.db.constants import SLACK_BOT_PERSONA_PREFIX
 from danswer.db.models import ChannelConfig
 from danswer.db.models import Persona
 from danswer.db.models import Persona__DocumentSet
+from danswer.db.models import SlackBotResponseType
 from danswer.db.models import SlackChannelConfig
 from danswer.db.models import User
 from danswer.db.persona import get_default_prompt
@@ -82,6 +83,7 @@ def insert_slack_channel_config(
     slack_bot_id: int,
     persona_id: int | None,
     channel_config: ChannelConfig,
+    response_type: SlackBotResponseType,
     standard_answer_category_ids: list[int],
     enable_auto_filters: bool,
 ) -> SlackChannelConfig:
@@ -113,6 +115,7 @@ def insert_slack_channel_config(
         slack_bot_id=slack_bot_id,
         persona_id=persona_id,
         channel_config=channel_config,
+        response_type=response_type,
         standard_answer_categories=existing_standard_answer_categories,
         enable_auto_filters=enable_auto_filters,
     )
@@ -127,6 +130,7 @@ def update_slack_channel_config(
     slack_channel_config_id: int,
     persona_id: int | None,
     channel_config: ChannelConfig,
+    response_type: SlackBotResponseType,
     standard_answer_category_ids: list[int],
     enable_auto_filters: bool,
 ) -> SlackChannelConfig:
@@ -166,6 +170,7 @@ def update_slack_channel_config(
     # will encounter `violates foreign key constraint` errors
     slack_channel_config.persona_id = persona_id
     slack_channel_config.channel_config = channel_config
+    slack_channel_config.response_type = response_type
     slack_channel_config.standard_answer_categories = list(
         existing_standard_answer_categories
     )

backend/danswer/document_index/interfaces.py

@@ -148,7 +148,6 @@ class Indexable(abc.ABC):
     def index(
         self,
         chunks: list[DocMetadataAwareIndexChunk],
-        fresh_index: bool = False,
     ) -> set[DocumentInsertionRecord]:
         """
         Takes a list of document chunks and indexes them in the document index
@@ -166,14 +165,9 @@ class Indexable(abc.ABC):
         only needs to index chunks into the PRIMARY index. Do not update the secondary index here,
         it is done automatically outside of this code.
 
-        NOTE: The fresh_index parameter, when set to True, assumes no documents have been previously
-        indexed for the given index/tenant. This can be used to optimize the indexing process for
-        new or empty indices.
-
         Parameters:
         - chunks: Document chunks with all of the information needed for indexing to the document
                 index.
-        - fresh_index: Boolean indicating whether this is a fresh index with no existing documents.
 
         Returns:
             List of document ids which map to unique documents and are used for deduping chunks

backend/danswer/document_index/vespa/app_config/schemas/danswer_chunk.sd

@@ -4,8 +4,6 @@ schema DANSWER_CHUNK_NAME {
         # Not to be confused with the UUID generated for this chunk which is called documentid by default
         field document_id type string {
             indexing: summary | attribute
-            attribute: fast-search
-            rank: filter
         }
         field chunk_id type int {
             indexing: summary | attribute

backend/danswer/document_index/vespa/index.py

@@ -306,7 +306,6 @@ class VespaIndex(DocumentIndex):
     def index(
         self,
         chunks: list[DocMetadataAwareIndexChunk],
-        fresh_index: bool = False,
     ) -> set[DocumentInsertionRecord]:
         """Receive a list of chunks from a batch of documents and index the chunks into Vespa along
         with updating the associated permissions. Assumes that a document will not be split into
@@ -323,29 +322,26 @@ class VespaIndex(DocumentIndex):
             concurrent.futures.ThreadPoolExecutor(max_workers=NUM_THREADS) as executor,
             get_vespa_http_client() as http_client,
         ):
-            if not fresh_index:
-                # Check for existing documents, existing documents need to have all of their chunks deleted
-                # prior to indexing as the document size (num chunks) may have shrunk
-                first_chunks = [
-                    chunk for chunk in cleaned_chunks if chunk.chunk_id == 0
-                ]
-                for chunk_batch in batch_generator(first_chunks, BATCH_SIZE):
-                    existing_docs.update(
-                        get_existing_documents_from_chunks(
-                            chunks=chunk_batch,
-                            index_name=self.index_name,
-                            http_client=http_client,
-                            executor=executor,
-                        )
-                    )
-
-                for doc_id_batch in batch_generator(existing_docs, BATCH_SIZE):
-                    delete_vespa_docs(
-                        document_ids=doc_id_batch,
+            # Check for existing documents, existing documents need to have all of their chunks deleted
+            # prior to indexing as the document size (num chunks) may have shrunk
+            first_chunks = [chunk for chunk in cleaned_chunks if chunk.chunk_id == 0]
+            for chunk_batch in batch_generator(first_chunks, BATCH_SIZE):
+                existing_docs.update(
+                    get_existing_documents_from_chunks(
+                        chunks=chunk_batch,
                         index_name=self.index_name,
                         http_client=http_client,
                         executor=executor,
                     )
+                )
+
+            for doc_id_batch in batch_generator(existing_docs, BATCH_SIZE):
+                delete_vespa_docs(
+                    document_ids=doc_id_batch,
+                    index_name=self.index_name,
+                    http_client=http_client,
+                    executor=executor,
+                )
 
             for chunk_batch in batch_generator(cleaned_chunks, BATCH_SIZE):
                 batch_index_vespa_chunks(

backend/danswer/file_processing/extract_file_text.py

@@ -6,7 +6,6 @@ import zipfile
 from collections.abc import Callable
 from collections.abc import Iterator
 from email.parser import Parser as EmailParser
-from io import BytesIO
 from pathlib import Path
 from typing import Any
 from typing import Dict
@@ -16,17 +15,13 @@ import chardet
 import docx  # type: ignore
 import openpyxl  # type: ignore
 import pptx  # type: ignore
-from docx import Document
-from fastapi import UploadFile
 from pypdf import PdfReader
 from pypdf.errors import PdfStreamError
 
 from danswer.configs.constants import DANSWER_METADATA_FILENAME
-from danswer.configs.constants import FileOrigin
 from danswer.file_processing.html_utils import parse_html_page_basic
 from danswer.file_processing.unstructured import get_unstructured_api_key
 from danswer.file_processing.unstructured import unstructured_to_text
-from danswer.file_store.file_store import FileStore
 from danswer.utils.logger import setup_logger
 
 logger = setup_logger()
@@ -70,7 +65,7 @@ def get_file_ext(file_path_or_name: str | Path) -> str:
     return extension
 
 
-def is_valid_file_ext(ext: str) -> bool:
+def check_file_ext_is_valid(ext: str) -> bool:
     return ext in VALID_FILE_EXTENSIONS
 
 
@@ -364,7 +359,7 @@ def extract_file_text(
             elif file_name is not None:
                 final_extension = get_file_ext(file_name)
 
-            if is_valid_file_ext(final_extension):
+            if check_file_ext_is_valid(final_extension):
                 return extension_to_function.get(final_extension, file_io_to_text)(file)
 
         # Either the file somehow has no name or the extension is not one that we recognize
@@ -380,35 +375,3 @@ def extract_file_text(
             ) from e
         logger.warning(f"Failed to process file {file_name or 'Unknown'}: {str(e)}")
         return ""
-
-
-def convert_docx_to_txt(
-    file: UploadFile, file_store: FileStore, file_path: str
-) -> None:
-    file.file.seek(0)
-    docx_content = file.file.read()
-    doc = Document(BytesIO(docx_content))
-
-    # Extract text from the document
-    full_text = []
-    for para in doc.paragraphs:
-        full_text.append(para.text)
-
-    # Join the extracted text
-    text_content = "\n".join(full_text)
-
-    txt_file_path = docx_to_txt_filename(file_path)
-    file_store.save_file(
-        file_name=txt_file_path,
-        content=BytesIO(text_content.encode("utf-8")),
-        display_name=file.filename,
-        file_origin=FileOrigin.CONNECTOR,
-        file_type="text/plain",
-    )
-
-
-def docx_to_txt_filename(file_path: str) -> str:
-    """
-    Convert a .docx file path to its corresponding .txt file path.
-    """
-    return file_path.rsplit(".", 1)[0] + ".txt"

backend/danswer/file_store/file_store.py

@@ -59,12 +59,6 @@ class FileStore(ABC):
             Contents of the file and metadata dict
         """
 
-    @abstractmethod
-    def read_file_record(self, file_name: str) -> PGFileStore:
-        """
-        Read the file record by the name
-        """
-
     @abstractmethod
     def delete_file(self, file_name: str) -> None:
         """

backend/danswer/file_store/utils.py

@@ -1,6 +1,6 @@
-import base64
 from collections.abc import Callable
 from io import BytesIO
+from typing import Any
 from typing import cast
 from uuid import uuid4
 
@@ -13,8 +13,8 @@ from danswer.db.models import ChatMessage
 from danswer.file_store.file_store import get_default_file_store
 from danswer.file_store.models import FileDescriptor
 from danswer.file_store.models import InMemoryChatFile
-from danswer.utils.b64 import get_image_type
 from danswer.utils.threadpool_concurrency import run_functions_tuples_in_parallel
+from shared_configs.contextvars import CURRENT_TENANT_ID_CONTEXTVAR
 
 
 def load_chat_file(
@@ -75,58 +75,11 @@ def save_file_from_url(url: str, tenant_id: str) -> str:
         return unique_id
 
 
-def save_file_from_base64(base64_string: str, tenant_id: str) -> str:
-    with get_session_with_tenant(tenant_id) as db_session:
-        unique_id = str(uuid4())
-        file_store = get_default_file_store(db_session)
-        file_store.save_file(
-            file_name=unique_id,
-            content=BytesIO(base64.b64decode(base64_string)),
-            display_name="GeneratedImage",
-            file_origin=FileOrigin.CHAT_IMAGE_GEN,
-            file_type=get_image_type(base64_string),
-        )
-        return unique_id
+def save_files_from_urls(urls: list[str]) -> list[str]:
+    tenant_id = CURRENT_TENANT_ID_CONTEXTVAR.get()
 
-
-def save_file(
-    tenant_id: str,
-    url: str | None = None,
-    base64_data: str | None = None,
-) -> str:
-    """Save a file from either a URL or base64 encoded string.
-
-    Args:
-        tenant_id: The tenant ID to save the file under
-        url: URL to download file from
-        base64_data: Base64 encoded file data
-
-    Returns:
-        The unique ID of the saved file
-
-    Raises:
-        ValueError: If neither url nor base64_data is provided, or if both are provided
-    """
-    if url is not None and base64_data is not None:
-        raise ValueError("Cannot specify both url and base64_data")
-
-    if url is not None:
-        return save_file_from_url(url, tenant_id)
-    elif base64_data is not None:
-        return save_file_from_base64(base64_data, tenant_id)
-    else:
-        raise ValueError("Must specify either url or base64_data")
-
-
-def save_files(urls: list[str], base64_files: list[str], tenant_id: str) -> list[str]:
-    # NOTE: be explicit about typing so that if we change things, we get notified
-    funcs: list[
-        tuple[
-            Callable[[str, str | None, str | None], str],
-            tuple[str, str | None, str | None],
-        ]
-    ] = [(save_file, (tenant_id, url, None)) for url in urls] + [
-        (save_file, (tenant_id, None, base64_file)) for base64_file in base64_files
+    funcs: list[tuple[Callable[..., Any], tuple[Any, ...]]] = [
+        (save_file_from_url, (url, tenant_id)) for url in urls
     ]
-
+    # Must pass in tenant_id here, since this is called by multithreading
     return run_functions_tuples_in_parallel(funcs)

backend/danswer/indexing/indexing_pipeline.py

@@ -1,5 +1,4 @@
 import traceback
-from collections.abc import Callable
 from functools import partial
 from http import HTTPStatus
 from typing import Protocol
@@ -13,7 +12,6 @@ from danswer.access.access import get_access_for_documents
 from danswer.access.models import DocumentAccess
 from danswer.configs.app_configs import ENABLE_MULTIPASS_INDEXING
 from danswer.configs.app_configs import INDEXING_EXCEPTION_LIMIT
-from danswer.configs.app_configs import MAX_DOCUMENT_CHARS
 from danswer.configs.constants import DEFAULT_BOOST
 from danswer.connectors.cross_connector_utils.miscellaneous_utils import (
     get_experts_stores_representations,
@@ -204,13 +202,40 @@ def index_doc_batch_with_handler(
 
 
 def index_doc_batch_prepare(
-    documents: list[Document],
+    document_batch: list[Document],
     index_attempt_metadata: IndexAttemptMetadata,
     db_session: Session,
     ignore_time_skip: bool = False,
 ) -> DocumentBatchPrepareContext | None:
     """Sets up the documents in the relational DB (source of truth) for permissions, metadata, etc.
     This preceeds indexing it into the actual document index."""
+    documents: list[Document] = []
+    for document in document_batch:
+        empty_contents = not any(section.text.strip() for section in document.sections)
+        if (
+            (not document.title or not document.title.strip())
+            and not document.semantic_identifier.strip()
+            and empty_contents
+        ):
+            # Skip documents that have neither title nor content
+            # If the document doesn't have either, then there is no useful information in it
+            # This is again verified later in the pipeline after chunking but at that point there should
+            # already be no documents that are empty.
+            logger.warning(
+                f"Skipping document with ID {document.id} as it has neither title nor content."
+            )
+            continue
+
+        if document.title is not None and not document.title.strip() and empty_contents:
+            # The title is explicitly empty ("" and not None) and the document is empty
+            # so when building the chunk text representation, it will be empty and unuseable
+            logger.warning(
+                f"Skipping document with ID {document.id} as the chunks will be empty."
+            )
+            continue
+
+        documents.append(document)
+
     # Create a trimmed list of docs that don't have a newer updated at
     # Shortcuts the time-consuming flow on connector index retries
     document_ids: list[str] = [document.id for document in documents]
@@ -257,64 +282,17 @@ def index_doc_batch_prepare(
     )
 
 
-def filter_documents(document_batch: list[Document]) -> list[Document]:
-    documents: list[Document] = []
-    for document in document_batch:
-        empty_contents = not any(section.text.strip() for section in document.sections)
-        if (
-            (not document.title or not document.title.strip())
-            and not document.semantic_identifier.strip()
-            and empty_contents
-        ):
-            # Skip documents that have neither title nor content
-            # If the document doesn't have either, then there is no useful information in it
-            # This is again verified later in the pipeline after chunking but at that point there should
-            # already be no documents that are empty.
-            logger.warning(
-                f"Skipping document with ID {document.id} as it has neither title nor content."
-            )
-            continue
-
-        if document.title is not None and not document.title.strip() and empty_contents:
-            # The title is explicitly empty ("" and not None) and the document is empty
-            # so when building the chunk text representation, it will be empty and unuseable
-            logger.warning(
-                f"Skipping document with ID {document.id} as the chunks will be empty."
-            )
-            continue
-
-        section_chars = sum(len(section.text) for section in document.sections)
-        if (
-            MAX_DOCUMENT_CHARS
-            and len(document.title or document.semantic_identifier) + section_chars
-            > MAX_DOCUMENT_CHARS
-        ):
-            # Skip documents that are too long, later on there are more memory intensive steps done on the text
-            # and the container will run out of memory and crash. Several other checks are included upstream but
-            # those are at the connector level so a catchall is still needed.
-            # Assumption here is that files that are that long, are generated files and not the type users
-            # generally care for.
-            logger.warning(
-                f"Skipping document with ID {document.id} as it is too long."
-            )
-            continue
-
-        documents.append(document)
-    return documents
-
-
 @log_function_time(debug_only=True)
 def index_doc_batch(
     *,
-    document_batch: list[Document],
     chunker: Chunker,
     embedder: IndexingEmbedder,
     document_index: DocumentIndex,
+    document_batch: list[Document],
     index_attempt_metadata: IndexAttemptMetadata,
     db_session: Session,
     ignore_time_skip: bool = False,
     tenant_id: str | None = None,
-    filter_fnc: Callable[[list[Document]], list[Document]] = filter_documents,
 ) -> tuple[int, int]:
     """Takes different pieces of the indexing pipeline and applies it to a batch of documents
     Note that the documents should already be batched at this point so that it does not inflate the
@@ -331,11 +309,8 @@ def index_doc_batch(
         is_public=False,
     )
 
-    logger.debug("Filtering Documents")
-    filtered_documents = filter_fnc(document_batch)
-
     ctx = index_doc_batch_prepare(
-        documents=filtered_documents,
+        document_batch=document_batch,
         index_attempt_metadata=index_attempt_metadata,
         ignore_time_skip=ignore_time_skip,
         db_session=db_session,

backend/danswer/llm/answering/answer.py

@@ -6,27 +6,33 @@ from langchain.schema.messages import BaseMessage
 from langchain_core.messages import AIMessageChunk
 from langchain_core.messages import ToolCall
 
-from danswer.chat.llm_response_handler import LLMResponseHandlerManager
 from danswer.chat.models import AnswerQuestionPossibleReturn
-from danswer.chat.models import AnswerStyleConfig
 from danswer.chat.models import CitationInfo
 from danswer.chat.models import DanswerAnswerPiece
-from danswer.chat.models import PromptConfig
-from danswer.chat.prompt_builder.build import AnswerPromptBuilder
-from danswer.chat.prompt_builder.build import default_build_system_message
-from danswer.chat.prompt_builder.build import default_build_user_message
-from danswer.chat.prompt_builder.build import LLMCall
-from danswer.chat.stream_processing.answer_response_handler import (
+from danswer.file_store.utils import InMemoryChatFile
+from danswer.llm.answering.llm_response_handler import LLMCall
+from danswer.llm.answering.llm_response_handler import LLMResponseHandlerManager
+from danswer.llm.answering.models import AnswerStyleConfig
+from danswer.llm.answering.models import PreviousMessage
+from danswer.llm.answering.models import PromptConfig
+from danswer.llm.answering.prompts.build import AnswerPromptBuilder
+from danswer.llm.answering.prompts.build import default_build_system_message
+from danswer.llm.answering.prompts.build import default_build_user_message
+from danswer.llm.answering.stream_processing.answer_response_handler import (
+    AnswerResponseHandler,
+)
+from danswer.llm.answering.stream_processing.answer_response_handler import (
     CitationResponseHandler,
 )
-from danswer.chat.stream_processing.answer_response_handler import (
+from danswer.llm.answering.stream_processing.answer_response_handler import (
     DummyAnswerResponseHandler,
 )
-from danswer.chat.stream_processing.utils import map_document_id_order
-from danswer.chat.tool_handling.tool_response_handler import ToolResponseHandler
-from danswer.file_store.utils import InMemoryChatFile
+from danswer.llm.answering.stream_processing.answer_response_handler import (
+    QuotesResponseHandler,
+)
+from danswer.llm.answering.stream_processing.utils import map_document_id_order
+from danswer.llm.answering.tool.tool_response_handler import ToolResponseHandler
 from danswer.llm.interfaces import LLM
-from danswer.llm.models import PreviousMessage
 from danswer.natural_language_processing.utils import get_tokenizer
 from danswer.tools.force import ForceUseTool
 from danswer.tools.models import ToolResponse
@@ -206,28 +212,20 @@ class Answer:
         # + figure out what the next LLM call should be
         tool_call_handler = ToolResponseHandler(current_llm_call.tools)
 
-        search_result, displayed_search_results_map = SearchTool.get_search_result(
-            current_llm_call
-        ) or ([], {})
+        search_result = SearchTool.get_search_result(current_llm_call) or []
 
-        # Quotes are no longer supported
-        # answer_handler: AnswerResponseHandler
-        # if self.answer_style_config.citation_config:
-        #     answer_handler = CitationResponseHandler(
-        #         context_docs=search_result,
-        #         doc_id_to_rank_map=map_document_id_order(search_result),
-        #     )
-        # elif self.answer_style_config.quotes_config:
-        #     answer_handler = QuotesResponseHandler(
-        #         context_docs=search_result,
-        #     )
-        # else:
-        #     raise ValueError("No answer style config provided")
-        answer_handler = CitationResponseHandler(
-            context_docs=search_result,
-            doc_id_to_rank_map=map_document_id_order(search_result),
-            display_doc_order_dict=displayed_search_results_map,
-        )
+        answer_handler: AnswerResponseHandler
+        if self.answer_style_config.citation_config:
+            answer_handler = CitationResponseHandler(
+                context_docs=search_result,
+                doc_id_to_rank_map=map_document_id_order(search_result),
+            )
+        elif self.answer_style_config.quotes_config:
+            answer_handler = QuotesResponseHandler(
+                context_docs=search_result,
+            )
+        else:
+            raise ValueError("No answer style config provided")
 
         response_handler_manager = LLMResponseHandlerManager(
             tool_call_handler, answer_handler, self.is_cancelled

backend/danswer/llm/answering/llm_response_handler.py

@@ -1,22 +1,60 @@
 from collections.abc import Callable
 from collections.abc import Generator
 from collections.abc import Iterator
+from typing import TYPE_CHECKING
 
 from langchain_core.messages import BaseMessage
+from pydantic.v1 import BaseModel as BaseModel__v1
 
-from danswer.chat.models import ResponsePart
+from danswer.chat.models import CitationInfo
+from danswer.chat.models import DanswerAnswerPiece
+from danswer.chat.models import DanswerQuotes
 from danswer.chat.models import StreamStopInfo
 from danswer.chat.models import StreamStopReason
-from danswer.chat.prompt_builder.build import LLMCall
-from danswer.chat.stream_processing.answer_response_handler import AnswerResponseHandler
-from danswer.chat.tool_handling.tool_response_handler import ToolResponseHandler
+from danswer.file_store.models import InMemoryChatFile
+from danswer.llm.answering.prompts.build import AnswerPromptBuilder
+from danswer.tools.force import ForceUseTool
+from danswer.tools.models import ToolCallFinalResult
+from danswer.tools.models import ToolCallKickoff
+from danswer.tools.models import ToolResponse
+from danswer.tools.tool import Tool
+
+
+if TYPE_CHECKING:
+    from danswer.llm.answering.stream_processing.answer_response_handler import (
+        AnswerResponseHandler,
+    )
+    from danswer.llm.answering.tool.tool_response_handler import ToolResponseHandler
+
+
+ResponsePart = (
+    DanswerAnswerPiece
+    | CitationInfo
+    | DanswerQuotes
+    | ToolCallKickoff
+    | ToolResponse
+    | ToolCallFinalResult
+    | StreamStopInfo
+)
+
+
+class LLMCall(BaseModel__v1):
+    prompt_builder: AnswerPromptBuilder
+    tools: list[Tool]
+    force_use_tool: ForceUseTool
+    files: list[InMemoryChatFile]
+    tool_call_info: list[ToolCallKickoff | ToolResponse | ToolCallFinalResult]
+    using_tool_calling_llm: bool
+
+    class Config:
+        arbitrary_types_allowed = True
 
 
 class LLMResponseHandlerManager:
     def __init__(
         self,
-        tool_handler: ToolResponseHandler,
-        answer_handler: AnswerResponseHandler,
+        tool_handler: "ToolResponseHandler",
+        answer_handler: "AnswerResponseHandler",
         is_cancelled: Callable[[], bool],
     ):
         self.tool_handler = tool_handler

backend/danswer/llm/answering/models.py

@@ -0,0 +1,163 @@
+from collections.abc import Callable
+from collections.abc import Iterator
+from typing import TYPE_CHECKING
+
+from langchain.schema.messages import AIMessage
+from langchain.schema.messages import BaseMessage
+from langchain.schema.messages import HumanMessage
+from langchain.schema.messages import SystemMessage
+from pydantic import BaseModel
+from pydantic import ConfigDict
+from pydantic import Field
+from pydantic import model_validator
+
+from danswer.chat.models import AnswerQuestionStreamReturn
+from danswer.configs.constants import MessageType
+from danswer.file_store.models import InMemoryChatFile
+from danswer.llm.override_models import PromptOverride
+from danswer.llm.utils import build_content_with_imgs
+from danswer.tools.models import ToolCallFinalResult
+
+if TYPE_CHECKING:
+    from danswer.db.models import ChatMessage
+    from danswer.db.models import Prompt
+
+
+StreamProcessor = Callable[[Iterator[str]], AnswerQuestionStreamReturn]
+
+
+class PreviousMessage(BaseModel):
+    """Simplified version of `ChatMessage`"""
+
+    message: str
+    token_count: int
+    message_type: MessageType
+    files: list[InMemoryChatFile]
+    tool_call: ToolCallFinalResult | None
+
+    @classmethod
+    def from_chat_message(
+        cls, chat_message: "ChatMessage", available_files: list[InMemoryChatFile]
+    ) -> "PreviousMessage":
+        message_file_ids = (
+            [file["id"] for file in chat_message.files] if chat_message.files else []
+        )
+        return cls(
+            message=chat_message.message,
+            token_count=chat_message.token_count,
+            message_type=chat_message.message_type,
+            files=[
+                file
+                for file in available_files
+                if str(file.file_id) in message_file_ids
+            ],
+            tool_call=ToolCallFinalResult(
+                tool_name=chat_message.tool_call.tool_name,
+                tool_args=chat_message.tool_call.tool_arguments,
+                tool_result=chat_message.tool_call.tool_result,
+            )
+            if chat_message.tool_call
+            else None,
+        )
+
+    def to_langchain_msg(self) -> BaseMessage:
+        content = build_content_with_imgs(self.message, self.files)
+        if self.message_type == MessageType.USER:
+            return HumanMessage(content=content)
+        elif self.message_type == MessageType.ASSISTANT:
+            return AIMessage(content=content)
+        else:
+            return SystemMessage(content=content)
+
+
+class DocumentPruningConfig(BaseModel):
+    max_chunks: int | None = None
+    max_window_percentage: float | None = None
+    max_tokens: int | None = None
+    # different pruning behavior is expected when the
+    # user manually selects documents they want to chat with
+    # e.g. we don't want to truncate each document to be no more
+    # than one chunk long
+    is_manually_selected_docs: bool = False
+    # If user specifies to include additional context Chunks for each match, then different pruning
+    # is used. As many Sections as possible are included, and the last Section is truncated
+    # If this is false, all of the Sections are truncated if they are longer than the expected Chunk size.
+    # Sections are often expected to be longer than the maximum Chunk size but Chunks should not be.
+    use_sections: bool = True
+    # If using tools, then we need to consider the tool length
+    tool_num_tokens: int = 0
+    # If using a tool message to represent the docs, then we have to JSON serialize
+    # the document content, which adds to the token count.
+    using_tool_message: bool = False
+
+
+class ContextualPruningConfig(DocumentPruningConfig):
+    num_chunk_multiple: int
+
+    @classmethod
+    def from_doc_pruning_config(
+        cls, num_chunk_multiple: int, doc_pruning_config: DocumentPruningConfig
+    ) -> "ContextualPruningConfig":
+        return cls(num_chunk_multiple=num_chunk_multiple, **doc_pruning_config.dict())
+
+
+class CitationConfig(BaseModel):
+    all_docs_useful: bool = False
+
+
+class QuotesConfig(BaseModel):
+    pass
+
+
+class AnswerStyleConfig(BaseModel):
+    citation_config: CitationConfig | None = None
+    quotes_config: QuotesConfig | None = None
+    document_pruning_config: DocumentPruningConfig = Field(
+        default_factory=DocumentPruningConfig
+    )
+    # forces the LLM to return a structured response, see
+    # https://platform.openai.com/docs/guides/structured-outputs/introduction
+    # right now, only used by the simple chat API
+    structured_response_format: dict | None = None
+
+    @model_validator(mode="after")
+    def check_quotes_and_citation(self) -> "AnswerStyleConfig":
+        if self.citation_config is None and self.quotes_config is None:
+            raise ValueError(
+                "One of `citation_config` or `quotes_config` must be provided"
+            )
+
+        if self.citation_config is not None and self.quotes_config is not None:
+            raise ValueError(
+                "Only one of `citation_config` or `quotes_config` must be provided"
+            )
+
+        return self
+
+
+class PromptConfig(BaseModel):
+    """Final representation of the Prompt configuration passed
+    into the `Answer` object."""
+
+    system_prompt: str
+    task_prompt: str
+    datetime_aware: bool
+    include_citations: bool
+
+    @classmethod
+    def from_model(
+        cls, model: "Prompt", prompt_override: PromptOverride | None = None
+    ) -> "PromptConfig":
+        override_system_prompt = (
+            prompt_override.system_prompt if prompt_override else None
+        )
+        override_task_prompt = prompt_override.task_prompt if prompt_override else None
+
+        return cls(
+            system_prompt=override_system_prompt or model.system_prompt,
+            task_prompt=override_task_prompt or model.task_prompt,
+            datetime_aware=model.datetime_aware,
+            include_citations=model.include_citations,
+        )
+
+    model_config = ConfigDict(frozen=True)

backend/danswer/llm/answering/prompts/build.py

@@ -4,26 +4,20 @@ from typing import cast
 from langchain_core.messages import BaseMessage
 from langchain_core.messages import HumanMessage
 from langchain_core.messages import SystemMessage
-from pydantic.v1 import BaseModel as BaseModel__v1
 
-from danswer.chat.models import PromptConfig
-from danswer.chat.prompt_builder.citations_prompt import compute_max_llm_input_tokens
-from danswer.chat.prompt_builder.utils import translate_history_to_basemessages
 from danswer.file_store.models import InMemoryChatFile
+from danswer.llm.answering.models import PreviousMessage
+from danswer.llm.answering.models import PromptConfig
+from danswer.llm.answering.prompts.citations_prompt import compute_max_llm_input_tokens
 from danswer.llm.interfaces import LLMConfig
-from danswer.llm.models import PreviousMessage
 from danswer.llm.utils import build_content_with_imgs
 from danswer.llm.utils import check_message_tokens
 from danswer.llm.utils import message_to_prompt_and_imgs
+from danswer.llm.utils import translate_history_to_basemessages
 from danswer.natural_language_processing.utils import get_tokenizer
 from danswer.prompts.chat_prompts import CHAT_USER_CONTEXT_FREE_PROMPT
 from danswer.prompts.prompt_utils import add_date_time_to_prompt
 from danswer.prompts.prompt_utils import drop_messages_history_overflow
-from danswer.tools.force import ForceUseTool
-from danswer.tools.models import ToolCallFinalResult
-from danswer.tools.models import ToolCallKickoff
-from danswer.tools.models import ToolResponse
-from danswer.tools.tool import Tool
 
 
 def default_build_system_message(
@@ -145,15 +139,3 @@ class AnswerPromptBuilder:
         return drop_messages_history_overflow(
             final_messages_with_tokens, self.max_tokens
         )
-
-
-class LLMCall(BaseModel__v1):
-    prompt_builder: AnswerPromptBuilder
-    tools: list[Tool]
-    force_use_tool: ForceUseTool
-    files: list[InMemoryChatFile]
-    tool_call_info: list[ToolCallKickoff | ToolResponse | ToolCallFinalResult]
-    using_tool_calling_llm: bool
-
-    class Config:
-        arbitrary_types_allowed = True

backend/danswer/llm/answering/prompts/citations_prompt.py

@@ -2,12 +2,12 @@ from langchain.schema.messages import HumanMessage
 from langchain.schema.messages import SystemMessage
 
 from danswer.chat.models import LlmDoc
-from danswer.chat.models import PromptConfig
 from danswer.configs.model_configs import GEN_AI_SINGLE_USER_MESSAGE_EXPECTED_MAX_TOKENS
 from danswer.context.search.models import InferenceChunk
 from danswer.db.models import Persona
 from danswer.db.persona import get_default_prompt__read_only
 from danswer.db.search_settings import get_multilingual_expansion
+from danswer.llm.answering.models import PromptConfig
 from danswer.llm.factory import get_llms_for_persona
 from danswer.llm.factory import get_main_llm_from_tuple
 from danswer.llm.interfaces import LLMConfig

backend/danswer/llm/answering/prompts/quotes_prompt.py

@@ -1,10 +1,10 @@
 from langchain.schema.messages import HumanMessage
 
 from danswer.chat.models import LlmDoc
-from danswer.chat.models import PromptConfig
 from danswer.configs.chat_configs import LANGUAGE_HINT
 from danswer.context.search.models import InferenceChunk
 from danswer.db.search_settings import get_multilingual_expansion
+from danswer.llm.answering.models import PromptConfig
 from danswer.llm.utils import message_to_prompt_and_imgs
 from danswer.prompts.direct_qa_prompts import CONTEXT_BLOCK
 from danswer.prompts.direct_qa_prompts import HISTORY_BLOCK

backend/danswer/llm/answering/prompts/utils.py

@@ -0,0 +1,20 @@
+from danswer.prompts.direct_qa_prompts import PARAMATERIZED_PROMPT
+from danswer.prompts.direct_qa_prompts import PARAMATERIZED_PROMPT_WITHOUT_CONTEXT
+
+
+def build_dummy_prompt(
+    system_prompt: str, task_prompt: str, retrieval_disabled: bool
+) -> str:
+    if retrieval_disabled:
+        return PARAMATERIZED_PROMPT_WITHOUT_CONTEXT.format(
+            user_query="<USER_QUERY>",
+            system_prompt=system_prompt,
+            task_prompt=task_prompt,
+        ).strip()
+
+    return PARAMATERIZED_PROMPT.format(
+        context_docs_str="<CONTEXT_DOCS>",
+        user_query="<USER_QUERY>",
+        system_prompt=system_prompt,
+        task_prompt=task_prompt,
+    ).strip()

backend/danswer/llm/answering/prune_and_merge.py

@@ -5,16 +5,16 @@ from typing import TypeVar
 
 from pydantic import BaseModel
 
-from danswer.chat.models import ContextualPruningConfig
 from danswer.chat.models import (
     LlmDoc,
 )
-from danswer.chat.models import PromptConfig
-from danswer.chat.prompt_builder.citations_prompt import compute_max_document_tokens
 from danswer.configs.constants import IGNORE_FOR_QA
 from danswer.configs.model_configs import DOC_EMBEDDING_CONTEXT_SIZE
 from danswer.context.search.models import InferenceChunk
 from danswer.context.search.models import InferenceSection
+from danswer.llm.answering.models import ContextualPruningConfig
+from danswer.llm.answering.models import PromptConfig
+from danswer.llm.answering.prompts.citations_prompt import compute_max_document_tokens
 from danswer.llm.interfaces import LLMConfig
 from danswer.natural_language_processing.utils import get_tokenizer
 from danswer.natural_language_processing.utils import tokenizer_trim_content

backend/danswer/llm/answering/stream_processing/answer_response_handler.py

@@ -3,11 +3,16 @@ from collections.abc import Generator
 
 from langchain_core.messages import BaseMessage
 
-from danswer.chat.llm_response_handler import ResponsePart
 from danswer.chat.models import CitationInfo
 from danswer.chat.models import LlmDoc
-from danswer.chat.stream_processing.citation_processing import CitationProcessor
-from danswer.chat.stream_processing.utils import DocumentIdOrderMapping
+from danswer.llm.answering.llm_response_handler import ResponsePart
+from danswer.llm.answering.stream_processing.citation_processing import (
+    CitationProcessor,
+)
+from danswer.llm.answering.stream_processing.quotes_processing import (
+    QuotesProcessor,
+)
+from danswer.llm.answering.stream_processing.utils import DocumentIdOrderMapping
 from danswer.utils.logger import setup_logger
 
 logger = setup_logger()
@@ -35,18 +40,13 @@ class DummyAnswerResponseHandler(AnswerResponseHandler):
 
 class CitationResponseHandler(AnswerResponseHandler):
     def __init__(
-        self,
-        context_docs: list[LlmDoc],
-        doc_id_to_rank_map: DocumentIdOrderMapping,
-        display_doc_order_dict: dict[str, int],
+        self, context_docs: list[LlmDoc], doc_id_to_rank_map: DocumentIdOrderMapping
     ):
         self.context_docs = context_docs
         self.doc_id_to_rank_map = doc_id_to_rank_map
-        self.display_doc_order_dict = display_doc_order_dict
         self.citation_processor = CitationProcessor(
             context_docs=self.context_docs,
             doc_id_to_rank_map=self.doc_id_to_rank_map,
-            display_doc_order_dict=self.display_doc_order_dict,
         )
         self.processed_text = ""
         self.citations: list[CitationInfo] = []
@@ -70,29 +70,28 @@ class CitationResponseHandler(AnswerResponseHandler):
         yield from self.citation_processor.process_token(content)
 
 
-# No longer in use, remove later
-# class QuotesResponseHandler(AnswerResponseHandler):
-#     def __init__(
-#         self,
-#         context_docs: list[LlmDoc],
-#         is_json_prompt: bool = True,
-#     ):
-#         self.quotes_processor = QuotesProcessor(
-#             context_docs=context_docs,
-#             is_json_prompt=is_json_prompt,
-#         )
+class QuotesResponseHandler(AnswerResponseHandler):
+    def __init__(
+        self,
+        context_docs: list[LlmDoc],
+        is_json_prompt: bool = True,
+    ):
+        self.quotes_processor = QuotesProcessor(
+            context_docs=context_docs,
+            is_json_prompt=is_json_prompt,
+        )
 
-#     def handle_response_part(
-#         self,
-#         response_item: BaseMessage | None,
-#         previous_response_items: list[BaseMessage],
-#     ) -> Generator[ResponsePart, None, None]:
-#         if response_item is None:
-#             yield from self.quotes_processor.process_token(None)
-#             return
+    def handle_response_part(
+        self,
+        response_item: BaseMessage | None,
+        previous_response_items: list[BaseMessage],
+    ) -> Generator[ResponsePart, None, None]:
+        if response_item is None:
+            yield from self.quotes_processor.process_token(None)
+            return
 
-#         content = (
-#             response_item.content if isinstance(response_item.content, str) else ""
-#         )
+        content = (
+            response_item.content if isinstance(response_item.content, str) else ""
+        )
 
-#         yield from self.quotes_processor.process_token(content)
+        yield from self.quotes_processor.process_token(content)

backend/danswer/llm/answering/stream_processing/citation_processing.py

@@ -4,8 +4,8 @@ from collections.abc import Generator
 from danswer.chat.models import CitationInfo
 from danswer.chat.models import DanswerAnswerPiece
 from danswer.chat.models import LlmDoc
-from danswer.chat.stream_processing.utils import DocumentIdOrderMapping
 from danswer.configs.chat_configs import STOP_STREAM_PAT
+from danswer.llm.answering.stream_processing.utils import DocumentIdOrderMapping
 from danswer.prompts.constants import TRIPLE_BACKTICK
 from danswer.utils.logger import setup_logger
 
@@ -22,16 +22,12 @@ class CitationProcessor:
         self,
         context_docs: list[LlmDoc],
         doc_id_to_rank_map: DocumentIdOrderMapping,
-        display_doc_order_dict: dict[str, int],
         stop_stream: str | None = STOP_STREAM_PAT,
     ):
         self.context_docs = context_docs
         self.doc_id_to_rank_map = doc_id_to_rank_map
         self.stop_stream = stop_stream
         self.order_mapping = doc_id_to_rank_map.order_mapping
-        self.display_doc_order_dict = (
-            display_doc_order_dict  # original order of docs to displayed to user
-        )
         self.llm_out = ""
         self.max_citation_num = len(context_docs)
         self.citation_order: list[int] = []
@@ -71,9 +67,9 @@ class CitationProcessor:
                 if piece_that_comes_after == "\n" and in_code_block(self.llm_out):
                     self.curr_segment = self.curr_segment.replace("```", "```plaintext")
 
-        citation_pattern = r"\[(\d+)\]|\[\[(\d+)\]\]"  # [1], [[1]], etc.
+        citation_pattern = r"\[(\d+)\]"
         citations_found = list(re.finditer(citation_pattern, self.curr_segment))
-        possible_citation_pattern = r"(\[+\d*$)"  # [1, [, [[, [[2, etc.
+        possible_citation_pattern = r"(\[\d*$)"  # [1, [, etc
         possible_citation_found = re.search(
             possible_citation_pattern, self.curr_segment
         )
@@ -81,15 +77,13 @@ class CitationProcessor:
         if len(citations_found) == 0 and len(self.llm_out) - self.past_cite_count > 5:
             self.current_citations = []
 
-        result = ""
+        result = ""  # Initialize result here
         if citations_found and not in_code_block(self.llm_out):
             last_citation_end = 0
             length_to_add = 0
             while len(citations_found) > 0:
                 citation = citations_found.pop(0)
-                numerical_value = int(
-                    next(group for group in citation.groups() if group is not None)
-                )
+                numerical_value = int(citation.group(1))
 
                 if 1 <= numerical_value <= self.max_citation_num:
                     context_llm_doc = self.context_docs[numerical_value - 1]
@@ -102,18 +96,6 @@ class CitationProcessor:
                         self.citation_order.index(real_citation_num) + 1
                     )
 
-                    # get the value that was displayed to user, should always
-                    # be in the display_doc_order_dict. But check anyways
-                    if context_llm_doc.document_id in self.display_doc_order_dict:
-                        displayed_citation_num = self.display_doc_order_dict[
-                            context_llm_doc.document_id
-                        ]
-                    else:
-                        displayed_citation_num = real_citation_num
-                        logger.warning(
-                            f"Doc {context_llm_doc.document_id} not in display_doc_order_dict. Used LLM citation number instead."
-                        )
-
                     # Skip consecutive citations of the same work
                     if target_citation_num in self.current_citations:
                         start, end = citation.span()
@@ -134,7 +116,6 @@ class CitationProcessor:
                                 doc_id = int(match.group(1))
                                 context_llm_doc = self.context_docs[doc_id - 1]
                                 yield CitationInfo(
-                                    # stay with the original for now (order of LLM cites)
                                     citation_num=target_citation_num,
                                     document_id=context_llm_doc.document_id,
                                 )
@@ -150,24 +131,29 @@ class CitationProcessor:
 
                     link = context_llm_doc.link
 
+                    # Replace the citation in the current segment
+                    start, end = citation.span()
+                    self.curr_segment = (
+                        self.curr_segment[: start + length_to_add]
+                        + f"[{target_citation_num}]"
+                        + self.curr_segment[end + length_to_add :]
+                    )
+
                     self.past_cite_count = len(self.llm_out)
                     self.current_citations.append(target_citation_num)
 
                     if target_citation_num not in self.cited_inds:
                         self.cited_inds.add(target_citation_num)
                         yield CitationInfo(
-                            # stay with the original for now (order of LLM cites)
                             citation_num=target_citation_num,
                             document_id=context_llm_doc.document_id,
                         )
 
-                    start, end = citation.span()
                     if link:
                         prev_length = len(self.curr_segment)
                         self.curr_segment = (
                             self.curr_segment[: start + length_to_add]
-                            + f"[[{displayed_citation_num}]]({link})"  # use the value that was displayed to user
-                            # + f"[[{target_citation_num}]]({link})"
+                            + f"[[{target_citation_num}]]({link})"
                             + self.curr_segment[end + length_to_add :]
                         )
                         length_to_add += len(self.curr_segment) - prev_length
@@ -175,8 +161,7 @@ class CitationProcessor:
                         prev_length = len(self.curr_segment)
                         self.curr_segment = (
                             self.curr_segment[: start + length_to_add]
-                            + f"[[{displayed_citation_num}]]()"  # use the value that was displayed to user
-                            # + f"[[{target_citation_num}]]()"
+                            + f"[[{target_citation_num}]]()"
                             + self.curr_segment[end + length_to_add :]
                         )
                         length_to_add += len(self.curr_segment) - prev_length

backend/danswer/llm/answering/stream_processing/quotes_processing.py

@@ -1,4 +1,3 @@
-# THIS IS NO LONGER IN USE
 import math
 import re
 from collections.abc import Generator
@@ -6,10 +5,11 @@ from json import JSONDecodeError
 from typing import Optional
 
 import regex
-from pydantic import BaseModel
 
 from danswer.chat.models import DanswerAnswer
 from danswer.chat.models import DanswerAnswerPiece
+from danswer.chat.models import DanswerQuote
+from danswer.chat.models import DanswerQuotes
 from danswer.chat.models import LlmDoc
 from danswer.configs.chat_configs import QUOTE_ALLOWED_ERROR_PERCENT
 from danswer.context.search.models import InferenceChunk
@@ -26,20 +26,6 @@ logger = setup_logger()
 answer_pattern = re.compile(r'{\s*"answer"\s*:\s*"', re.IGNORECASE)
 
 
-class DanswerQuote(BaseModel):
-    # This is during inference so everything is a string by this point
-    quote: str
-    document_id: str
-    link: str | None
-    source_type: str
-    semantic_identifier: str
-    blurb: str
-
-
-class DanswerQuotes(BaseModel):
-    quotes: list[DanswerQuote]
-
-
 def _extract_answer_quotes_freeform(
     answer_raw: str,
 ) -> tuple[Optional[str], Optional[list[str]]]:

backend/danswer/llm/answering/tool/tool_response_handler.py

@@ -4,8 +4,8 @@ from langchain_core.messages import AIMessageChunk
 from langchain_core.messages import BaseMessage
 from langchain_core.messages import ToolCall
 
-from danswer.chat.models import ResponsePart
-from danswer.chat.prompt_builder.build import LLMCall
+from danswer.llm.answering.llm_response_handler import LLMCall
+from danswer.llm.answering.llm_response_handler import ResponsePart
 from danswer.llm.interfaces import LLM
 from danswer.tools.force import ForceUseTool
 from danswer.tools.message import build_tool_message

backend/danswer/llm/chat_llm.py

@@ -268,16 +268,12 @@ class DefaultMultiLLM(LLM):
 
         # NOTE: have to set these as environment variables for Litellm since
         # not all are able to passed in but they always support them set as env
-        # variables. We'll also try passing them in, since litellm just ignores
-        # addtional kwargs (and some kwargs MUST be passed in rather than set as
-        # env variables)
+        # variables
         if custom_config:
             for k, v in custom_config.items():
                 os.environ[k] = v
 
         model_kwargs = model_kwargs or {}
-        if custom_config:
-            model_kwargs.update(custom_config)
         if extra_headers:
             model_kwargs.update({"extra_headers": extra_headers})
         if extra_body:

backend/danswer/llm/factory.py

@@ -1,6 +1,5 @@
 from typing import Any
 
-from danswer.chat.models import PersonaOverrideConfig
 from danswer.configs.app_configs import DISABLE_GENERATIVE_AI
 from danswer.configs.chat_configs import QA_TIMEOUT
 from danswer.configs.model_configs import GEN_AI_MODEL_FALLBACK_MAX_TOKENS
@@ -14,11 +13,8 @@ from danswer.llm.exceptions import GenAIDisabledException
 from danswer.llm.interfaces import LLM
 from danswer.llm.override_models import LLMOverride
 from danswer.utils.headers import build_llm_extra_headers
-from danswer.utils.logger import setup_logger
 from danswer.utils.long_term_log import LongTermLogger
 
-logger = setup_logger()
-
 
 def _build_extra_model_kwargs(provider: str) -> dict[str, Any]:
     """Ollama requires us to specify the max context window.
@@ -36,15 +32,11 @@ def get_main_llm_from_tuple(
 
 
 def get_llms_for_persona(
-    persona: Persona | PersonaOverrideConfig | None,
+    persona: Persona,
     llm_override: LLMOverride | None = None,
     additional_headers: dict[str, str] | None = None,
     long_term_logger: LongTermLogger | None = None,
 ) -> tuple[LLM, LLM]:
-    if persona is None:
-        logger.warning("No persona provided, using default LLMs")
-        return get_default_llms()
-
     model_provider_override = llm_override.model_provider if llm_override else None
     model_version_override = llm_override.model_version if llm_override else None
     temperature_override = llm_override.temperature if llm_override else None
@@ -79,7 +71,6 @@ def get_llms_for_persona(
             api_base=llm_provider.api_base,
             api_version=llm_provider.api_version,
             custom_config=llm_provider.custom_config,
-            temperature=temperature_override,
             additional_headers=additional_headers,
             long_term_logger=long_term_logger,
         )
@@ -137,13 +128,11 @@ def get_llm(
     api_base: str | None = None,
     api_version: str | None = None,
     custom_config: dict[str, str] | None = None,
-    temperature: float | None = None,
+    temperature: float = GEN_AI_TEMPERATURE,
     timeout: int = QA_TIMEOUT,
     additional_headers: dict[str, str] | None = None,
     long_term_logger: LongTermLogger | None = None,
 ) -> LLM:
-    if temperature is None:
-        temperature = GEN_AI_TEMPERATURE
     return DefaultMultiLLM(
         model_provider=provider,
         model_name=model,

backend/danswer/llm/models.py

@@ -1,59 +0,0 @@
-from typing import TYPE_CHECKING
-
-from langchain.schema.messages import AIMessage
-from langchain.schema.messages import BaseMessage
-from langchain.schema.messages import HumanMessage
-from langchain.schema.messages import SystemMessage
-from pydantic import BaseModel
-
-from danswer.configs.constants import MessageType
-from danswer.file_store.models import InMemoryChatFile
-from danswer.llm.utils import build_content_with_imgs
-from danswer.tools.models import ToolCallFinalResult
-
-if TYPE_CHECKING:
-    from danswer.db.models import ChatMessage
-
-
-class PreviousMessage(BaseModel):
-    """Simplified version of `ChatMessage`"""
-
-    message: str
-    token_count: int
-    message_type: MessageType
-    files: list[InMemoryChatFile]
-    tool_call: ToolCallFinalResult | None
-
-    @classmethod
-    def from_chat_message(
-        cls, chat_message: "ChatMessage", available_files: list[InMemoryChatFile]
-    ) -> "PreviousMessage":
-        message_file_ids = (
-            [file["id"] for file in chat_message.files] if chat_message.files else []
-        )
-        return cls(
-            message=chat_message.message,
-            token_count=chat_message.token_count,
-            message_type=chat_message.message_type,
-            files=[
-                file
-                for file in available_files
-                if str(file.file_id) in message_file_ids
-            ],
-            tool_call=ToolCallFinalResult(
-                tool_name=chat_message.tool_call.tool_name,
-                tool_args=chat_message.tool_call.tool_arguments,
-                tool_result=chat_message.tool_call.tool_result,
-            )
-            if chat_message.tool_call
-            else None,
-        )
-
-    def to_langchain_msg(self) -> BaseMessage:
-        content = build_content_with_imgs(self.message, self.files)
-        if self.message_type == MessageType.USER:
-            return HumanMessage(content=content)
-        elif self.message_type == MessageType.ASSISTANT:
-            return AIMessage(content=content)
-        else:
-            return SystemMessage(content=content)

backend/danswer/llm/utils.py

@@ -1,11 +1,15 @@
 import copy
+import io
 import json
 from collections.abc import Callable
 from collections.abc import Iterator
 from typing import Any
 from typing import cast
+from typing import TYPE_CHECKING
+from typing import Union
 
 import litellm  # type: ignore
+import pandas as pd
 import tiktoken
 from langchain.prompts.base import StringPromptValue
 from langchain.prompts.chat import ChatPromptValue
@@ -32,15 +36,17 @@ from danswer.configs.constants import MessageType
 from danswer.configs.model_configs import GEN_AI_MAX_TOKENS
 from danswer.configs.model_configs import GEN_AI_MODEL_FALLBACK_MAX_TOKENS
 from danswer.configs.model_configs import GEN_AI_NUM_RESERVED_OUTPUT_TOKENS
+from danswer.db.models import ChatMessage
 from danswer.file_store.models import ChatFileType
 from danswer.file_store.models import InMemoryChatFile
 from danswer.llm.interfaces import LLM
 from danswer.prompts.constants import CODE_BLOCK_PAT
-from danswer.utils.b64 import get_image_type
-from danswer.utils.b64 import get_image_type_from_bytes
 from danswer.utils.logger import setup_logger
 from shared_configs.configs import LOG_LEVEL
 
+if TYPE_CHECKING:
+    from danswer.llm.answering.models import PreviousMessage
+
 logger = setup_logger()
 
 
@@ -98,39 +104,92 @@ def litellm_exception_to_error_msg(
     return error_msg
 
 
+def translate_danswer_msg_to_langchain(
+    msg: Union[ChatMessage, "PreviousMessage"],
+) -> BaseMessage:
+    files: list[InMemoryChatFile] = []
+
+    # If the message is a `ChatMessage`, it doesn't have the downloaded files
+    # attached. Just ignore them for now.
+    if not isinstance(msg, ChatMessage):
+        files = msg.files
+    content = build_content_with_imgs(msg.message, files, message_type=msg.message_type)
+
+    if msg.message_type == MessageType.SYSTEM:
+        raise ValueError("System messages are not currently part of history")
+    if msg.message_type == MessageType.ASSISTANT:
+        return AIMessage(content=content)
+    if msg.message_type == MessageType.USER:
+        return HumanMessage(content=content)
+
+    raise ValueError(f"New message type {msg.message_type} not handled")
+
+
+def translate_history_to_basemessages(
+    history: list[ChatMessage] | list["PreviousMessage"],
+) -> tuple[list[BaseMessage], list[int]]:
+    history_basemessages = [
+        translate_danswer_msg_to_langchain(msg)
+        for msg in history
+        if msg.token_count != 0
+    ]
+    history_token_counts = [msg.token_count for msg in history if msg.token_count != 0]
+    return history_basemessages, history_token_counts
+
+
+#  Processes CSV files to show the first 5 rows and max_columns (default 40) columns
+def _process_csv_file(file: InMemoryChatFile, max_columns: int = 40) -> str:
+    df = pd.read_csv(io.StringIO(file.content.decode("utf-8")))
+
+    csv_preview = df.head().to_string(max_cols=max_columns)
+
+    file_name_section = (
+        f"CSV FILE NAME: {file.filename}\n"
+        if file.filename
+        else "CSV FILE (NO NAME PROVIDED):\n"
+    )
+    return f"{file_name_section}{CODE_BLOCK_PAT.format(csv_preview)}\n\n\n"
+
+
 def _build_content(
     message: str,
     files: list[InMemoryChatFile] | None = None,
 ) -> str:
     """Applies all non-image files."""
-    if not files:
-        return message
+    text_files = (
+        [file for file in files if file.file_type == ChatFileType.PLAIN_TEXT]
+        if files
+        else None
+    )
 
-    text_files = [
-        file
-        for file in files
-        if file.file_type in (ChatFileType.PLAIN_TEXT, ChatFileType.CSV)
-    ]
+    csv_files = (
+        [file for file in files if file.file_type == ChatFileType.CSV]
+        if files
+        else None
+    )
 
-    if not text_files:
+    if not text_files and not csv_files:
         return message
 
     final_message_with_files = "FILES:\n\n"
-    for file in text_files:
+    for file in text_files or []:
         file_content = file.content.decode("utf-8")
         file_name_section = f"DOCUMENT: {file.filename}\n" if file.filename else ""
         final_message_with_files += (
             f"{file_name_section}{CODE_BLOCK_PAT.format(file_content.strip())}\n\n\n"
         )
+    for file in csv_files or []:
+        final_message_with_files += _process_csv_file(file)
 
-    return final_message_with_files + message
+    final_message_with_files += message
+
+    return final_message_with_files
 
 
 def build_content_with_imgs(
     message: str,
     files: list[InMemoryChatFile] | None = None,
     img_urls: list[str] | None = None,
-    b64_imgs: list[str] | None = None,
     message_type: MessageType = MessageType.USER,
 ) -> str | list[str | dict[str, Any]]:  # matching Langchain's BaseMessage content type
     files = files or []
@@ -143,7 +202,6 @@ def build_content_with_imgs(
     )
 
     img_urls = img_urls or []
-    b64_imgs = b64_imgs or []
 
     message_main_content = _build_content(message, files)
 
@@ -162,22 +220,11 @@ def build_content_with_imgs(
             {
                 "type": "image_url",
                 "image_url": {
-                    "url": (
-                        f"data:{get_image_type_from_bytes(file.content)};"
-                        f"base64,{file.to_base64()}"
-                    ),
+                    "url": f"data:image/jpeg;base64,{file.to_base64()}",
                 },
             }
-            for file in img_files
-        ]
-        + [
-            {
-                "type": "image_url",
-                "image_url": {
-                    "url": f"data:{get_image_type(b64_img)};base64,{b64_img}",
-                },
-            }
-            for b64_img in b64_imgs
+            for file in files
+            if file.file_type == "image"
         ]
         + [
             {

backend/danswer/main.py

@@ -25,6 +25,7 @@ from danswer.auth.schemas import UserCreate
 from danswer.auth.schemas import UserRead
 from danswer.auth.schemas import UserUpdate
 from danswer.auth.users import auth_backend
+from danswer.auth.users import BasicAuthenticationError
 from danswer.auth.users import create_danswer_oauth_router
 from danswer.auth.users import fastapi_users
 from danswer.configs.app_configs import APP_API_PREFIX
@@ -52,9 +53,12 @@ from danswer.server.documents.connector import router as connector_router
 from danswer.server.documents.credential import router as credential_router
 from danswer.server.documents.document import router as document_router
 from danswer.server.documents.indexing import router as indexing_router
-from danswer.server.documents.standard_oauth import router as oauth_router
 from danswer.server.features.document_set.api import router as document_set_router
 from danswer.server.features.folder.api import router as folder_router
+from danswer.server.features.input_prompt.api import (
+    admin_router as admin_input_prompt_router,
+)
+from danswer.server.features.input_prompt.api import basic_router as input_prompt_router
 from danswer.server.features.notifications.api import router as notification_router
 from danswer.server.features.persona.api import admin_router as admin_persona_router
 from danswer.server.features.persona.api import basic_router as persona_router
@@ -88,7 +92,6 @@ from danswer.server.settings.api import basic_router as settings_router
 from danswer.server.token_rate_limits.api import (
     router as token_rate_limit_settings_router,
 )
-from danswer.server.utils import BasicAuthenticationError
 from danswer.setup import setup_danswer
 from danswer.setup import setup_multitenant_danswer
 from danswer.utils.logger import setup_logger
@@ -102,6 +105,7 @@ from shared_configs.configs import CORS_ALLOWED_ORIGIN
 from shared_configs.configs import MULTI_TENANT
 from shared_configs.configs import SENTRY_DSN
 
+
 logger = setup_logger()
 
 
@@ -202,7 +206,7 @@ def log_http_error(_: Request, exc: Exception) -> JSONResponse:
 
     if isinstance(exc, BasicAuthenticationError):
         # For BasicAuthenticationError, just log a brief message without stack trace (almost always spam)
-        logger.warning(f"Authentication failed: {str(exc)}")
+        logger.error(f"Authentication failed: {str(exc)}")
 
     elif status_code >= 400:
         error_msg = f"{str(exc)}\n"
@@ -255,6 +259,8 @@ def get_application() -> FastAPI:
     )
     include_router_with_global_prefix_prepended(application, persona_router)
     include_router_with_global_prefix_prepended(application, admin_persona_router)
+    include_router_with_global_prefix_prepended(application, input_prompt_router)
+    include_router_with_global_prefix_prepended(application, admin_input_prompt_router)
     include_router_with_global_prefix_prepended(application, notification_router)
     include_router_with_global_prefix_prepended(application, prompt_router)
     include_router_with_global_prefix_prepended(application, tool_router)
@@ -277,7 +283,6 @@ def get_application() -> FastAPI:
     )
     include_router_with_global_prefix_prepended(application, long_term_logs_router)
     include_router_with_global_prefix_prepended(application, api_key_router)
-    include_router_with_global_prefix_prepended(application, oauth_router)
 
     if AUTH_TYPE == AuthType.DISABLED:
         # Server logs this during auth setup verification step

backend/danswer/one_shot_answer/answer_question.py

@@ -0,0 +1,456 @@
+from collections.abc import Callable
+from collections.abc import Iterator
+from typing import cast
+
+from sqlalchemy.orm import Session
+
+from danswer.chat.chat_utils import reorganize_citations
+from danswer.chat.models import CitationInfo
+from danswer.chat.models import DanswerAnswerPiece
+from danswer.chat.models import DanswerContexts
+from danswer.chat.models import DanswerQuotes
+from danswer.chat.models import DocumentRelevance
+from danswer.chat.models import LLMRelevanceFilterResponse
+from danswer.chat.models import QADocsResponse
+from danswer.chat.models import RelevanceAnalysis
+from danswer.chat.models import StreamingError
+from danswer.configs.chat_configs import DISABLE_LLM_DOC_RELEVANCE
+from danswer.configs.chat_configs import MAX_CHUNKS_FED_TO_CHAT
+from danswer.configs.chat_configs import QA_TIMEOUT
+from danswer.configs.constants import MessageType
+from danswer.context.search.enums import LLMEvaluationType
+from danswer.context.search.models import RerankMetricsContainer
+from danswer.context.search.models import RetrievalMetricsContainer
+from danswer.context.search.utils import chunks_or_sections_to_search_docs
+from danswer.context.search.utils import dedupe_documents
+from danswer.db.chat import create_chat_session
+from danswer.db.chat import create_db_search_doc
+from danswer.db.chat import create_new_chat_message
+from danswer.db.chat import get_or_create_root_message
+from danswer.db.chat import translate_db_message_to_chat_message_detail
+from danswer.db.chat import translate_db_search_doc_to_server_search_doc
+from danswer.db.chat import update_search_docs_table_with_relevance
+from danswer.db.engine import get_session_context_manager
+from danswer.db.models import Persona
+from danswer.db.models import User
+from danswer.db.persona import get_prompt_by_id
+from danswer.llm.answering.answer import Answer
+from danswer.llm.answering.models import AnswerStyleConfig
+from danswer.llm.answering.models import CitationConfig
+from danswer.llm.answering.models import DocumentPruningConfig
+from danswer.llm.answering.models import PromptConfig
+from danswer.llm.answering.models import QuotesConfig
+from danswer.llm.factory import get_llms_for_persona
+from danswer.llm.factory import get_main_llm_from_tuple
+from danswer.natural_language_processing.utils import get_tokenizer
+from danswer.one_shot_answer.models import DirectQARequest
+from danswer.one_shot_answer.models import OneShotQAResponse
+from danswer.one_shot_answer.models import QueryRephrase
+from danswer.one_shot_answer.qa_utils import combine_message_thread
+from danswer.one_shot_answer.qa_utils import slackify_message_thread
+from danswer.secondary_llm_flows.answer_validation import get_answer_validity
+from danswer.secondary_llm_flows.query_expansion import thread_based_query_rephrase
+from danswer.server.query_and_chat.models import ChatMessageDetail
+from danswer.server.utils import get_json_line
+from danswer.tools.force import ForceUseTool
+from danswer.tools.models import ToolResponse
+from danswer.tools.tool_implementations.search.search_tool import SEARCH_DOC_CONTENT_ID
+from danswer.tools.tool_implementations.search.search_tool import (
+    SEARCH_RESPONSE_SUMMARY_ID,
+)
+from danswer.tools.tool_implementations.search.search_tool import SearchResponseSummary
+from danswer.tools.tool_implementations.search.search_tool import SearchTool
+from danswer.tools.tool_implementations.search.search_tool import (
+    SECTION_RELEVANCE_LIST_ID,
+)
+from danswer.tools.tool_runner import ToolCallKickoff
+from danswer.utils.logger import setup_logger
+from danswer.utils.long_term_log import LongTermLogger
+from danswer.utils.timing import log_generator_function_time
+from danswer.utils.variable_functionality import fetch_ee_implementation_or_noop
+
+logger = setup_logger()
+
+AnswerObjectIterator = Iterator[
+    QueryRephrase
+    | QADocsResponse
+    | LLMRelevanceFilterResponse
+    | DanswerAnswerPiece
+    | DanswerQuotes
+    | DanswerContexts
+    | StreamingError
+    | ChatMessageDetail
+    | CitationInfo
+    | ToolCallKickoff
+    | DocumentRelevance
+]
+
+
+def stream_answer_objects(
+    query_req: DirectQARequest,
+    user: User | None,
+    # These need to be passed in because in Web UI one shot flow,
+    # we can have much more document as there is no history.
+    # For Slack flow, we need to save more tokens for the thread context
+    max_document_tokens: int | None,
+    max_history_tokens: int | None,
+    db_session: Session,
+    # Needed to translate persona num_chunks to tokens to the LLM
+    default_num_chunks: float = MAX_CHUNKS_FED_TO_CHAT,
+    timeout: int = QA_TIMEOUT,
+    bypass_acl: bool = False,
+    use_citations: bool = False,
+    danswerbot_flow: bool = False,
+    retrieval_metrics_callback: (
+        Callable[[RetrievalMetricsContainer], None] | None
+    ) = None,
+    rerank_metrics_callback: Callable[[RerankMetricsContainer], None] | None = None,
+) -> AnswerObjectIterator:
+    """Streams in order:
+    1. [always] Retrieved documents, stops flow if nothing is found
+    2. [conditional] LLM selected chunk indices if LLM chunk filtering is turned on
+    3. [always] A set of streamed DanswerAnswerPiece and DanswerQuotes at the end
+                or an error anywhere along the line if something fails
+    4. [always] Details on the final AI response message that is created
+    """
+    user_id = user.id if user is not None else None
+    query_msg = query_req.messages[-1]
+    history = query_req.messages[:-1]
+
+    chat_session = create_chat_session(
+        db_session=db_session,
+        description="",  # One shot queries don't need naming as it's never displayed
+        user_id=user_id,
+        persona_id=query_req.persona_id,
+        one_shot=True,
+        danswerbot_flow=danswerbot_flow,
+    )
+
+    # permanent "log" store, used primarily for debugging
+    long_term_logger = LongTermLogger(
+        metadata={"user_id": str(user_id), "chat_session_id": str(chat_session.id)}
+    )
+
+    temporary_persona: Persona | None = None
+
+    if query_req.persona_config is not None:
+        temporary_persona = fetch_ee_implementation_or_noop(
+            "danswer.server.query_and_chat.utils", "create_temporary_persona", None
+        )(db_session=db_session, persona_config=query_req.persona_config, user=user)
+
+    persona = temporary_persona if temporary_persona else chat_session.persona
+
+    try:
+        llm, fast_llm = get_llms_for_persona(
+            persona=persona, long_term_logger=long_term_logger
+        )
+    except ValueError as e:
+        logger.error(
+            f"Failed to initialize LLMs for persona '{persona.name}': {str(e)}"
+        )
+        if "No LLM provider" in str(e):
+            raise ValueError(
+                "Please configure a Generative AI model to use this feature."
+            ) from e
+        raise ValueError(
+            "Failed to initialize the AI model. Please check your configuration and try again."
+        ) from e
+
+    llm_tokenizer = get_tokenizer(
+        model_name=llm.config.model_name,
+        provider_type=llm.config.model_provider,
+    )
+
+    # Create a chat session which will just store the root message, the query, and the AI response
+    root_message = get_or_create_root_message(
+        chat_session_id=chat_session.id, db_session=db_session
+    )
+
+    history_str = combine_message_thread(
+        messages=history,
+        max_tokens=max_history_tokens,
+        llm_tokenizer=llm_tokenizer,
+    )
+
+    rephrased_query = query_req.query_override or thread_based_query_rephrase(
+        user_query=query_msg.message,
+        history_str=history_str,
+    )
+
+    # Given back ahead of the documents for latency reasons
+    # In chat flow it's given back along with the documents
+    yield QueryRephrase(rephrased_query=rephrased_query)
+
+    prompt = None
+    if query_req.prompt_id is not None:
+        # NOTE: let the user access any prompt as long as the Persona is shared
+        # with them
+        prompt = get_prompt_by_id(
+            prompt_id=query_req.prompt_id, user=None, db_session=db_session
+        )
+    if prompt is None:
+        if not persona.prompts:
+            raise RuntimeError(
+                "Persona does not have any prompts - this should never happen"
+            )
+        prompt = persona.prompts[0]
+
+    user_message_str = query_msg.message
+    # For this endpoint, we only save one user message to the chat session
+    # However, for slackbot, we want to include the history of the entire thread
+    if danswerbot_flow:
+        # Right now, we only support bringing over citations and search docs
+        # from the last message in the thread, not the entire thread
+        # in the future, we may want to retrieve the entire thread
+        user_message_str = slackify_message_thread(query_req.messages)
+
+    # Create the first User query message
+    new_user_message = create_new_chat_message(
+        chat_session_id=chat_session.id,
+        parent_message=root_message,
+        prompt_id=query_req.prompt_id,
+        message=user_message_str,
+        token_count=len(llm_tokenizer.encode(user_message_str)),
+        message_type=MessageType.USER,
+        db_session=db_session,
+        commit=True,
+    )
+
+    prompt_config = PromptConfig.from_model(prompt)
+    document_pruning_config = DocumentPruningConfig(
+        max_chunks=int(
+            persona.num_chunks if persona.num_chunks is not None else default_num_chunks
+        ),
+        max_tokens=max_document_tokens,
+    )
+
+    answer_config = AnswerStyleConfig(
+        citation_config=CitationConfig() if use_citations else None,
+        quotes_config=QuotesConfig() if not use_citations else None,
+        document_pruning_config=document_pruning_config,
+    )
+
+    search_tool = SearchTool(
+        db_session=db_session,
+        user=user,
+        evaluation_type=(
+            LLMEvaluationType.SKIP
+            if DISABLE_LLM_DOC_RELEVANCE
+            else query_req.evaluation_type
+        ),
+        persona=persona,
+        retrieval_options=query_req.retrieval_options,
+        prompt_config=prompt_config,
+        llm=llm,
+        fast_llm=fast_llm,
+        pruning_config=document_pruning_config,
+        answer_style_config=answer_config,
+        bypass_acl=bypass_acl,
+        chunks_above=query_req.chunks_above,
+        chunks_below=query_req.chunks_below,
+        full_doc=query_req.full_doc,
+    )
+
+    answer = Answer(
+        question=query_msg.message,
+        answer_style_config=answer_config,
+        prompt_config=PromptConfig.from_model(prompt),
+        llm=get_main_llm_from_tuple(
+            get_llms_for_persona(persona=persona, long_term_logger=long_term_logger)
+        ),
+        single_message_history=history_str,
+        tools=[search_tool] if search_tool else [],
+        force_use_tool=(
+            ForceUseTool(
+                tool_name=search_tool.name,
+                args={"query": rephrased_query},
+                force_use=True,
+            )
+        ),
+        # for now, don't use tool calling for this flow, as we haven't
+        # tested quotes with tool calling too much yet
+        skip_explicit_tool_calling=True,
+        return_contexts=query_req.return_contexts,
+        skip_gen_ai_answer_generation=query_req.skip_gen_ai_answer_generation,
+    )
+    # won't be any FileChatDisplay responses since that tool is never passed in
+    for packet in cast(AnswerObjectIterator, answer.processed_streamed_output):
+        # for one-shot flow, don't currently do anything with these
+        if isinstance(packet, ToolResponse):
+            # (likely fine that it comes after the initial creation of the search docs)
+            if packet.id == SEARCH_RESPONSE_SUMMARY_ID:
+                search_response_summary = cast(SearchResponseSummary, packet.response)
+
+                top_docs = chunks_or_sections_to_search_docs(
+                    search_response_summary.top_sections
+                )
+
+                # Deduping happens at the last step to avoid harming quality by dropping content early on
+                deduped_docs = top_docs
+                if query_req.retrieval_options.dedupe_docs:
+                    deduped_docs, dropped_inds = dedupe_documents(top_docs)
+
+                reference_db_search_docs = [
+                    create_db_search_doc(server_search_doc=doc, db_session=db_session)
+                    for doc in deduped_docs
+                ]
+
+                response_docs = [
+                    translate_db_search_doc_to_server_search_doc(db_search_doc)
+                    for db_search_doc in reference_db_search_docs
+                ]
+
+                initial_response = QADocsResponse(
+                    rephrased_query=rephrased_query,
+                    top_documents=response_docs,
+                    predicted_flow=search_response_summary.predicted_flow,
+                    predicted_search=search_response_summary.predicted_search,
+                    applied_source_filters=search_response_summary.final_filters.source_type,
+                    applied_time_cutoff=search_response_summary.final_filters.time_cutoff,
+                    recency_bias_multiplier=search_response_summary.recency_bias_multiplier,
+                )
+
+                yield initial_response
+
+            elif packet.id == SEARCH_DOC_CONTENT_ID:
+                yield packet.response
+
+            elif packet.id == SECTION_RELEVANCE_LIST_ID:
+                document_based_response = {}
+
+                if packet.response is not None:
+                    for evaluation in packet.response:
+                        document_based_response[
+                            evaluation.document_id
+                        ] = RelevanceAnalysis(
+                            relevant=evaluation.relevant, content=evaluation.content
+                        )
+
+                evaluation_response = DocumentRelevance(
+                    relevance_summaries=document_based_response
+                )
+                if reference_db_search_docs is not None:
+                    update_search_docs_table_with_relevance(
+                        db_session=db_session,
+                        reference_db_search_docs=reference_db_search_docs,
+                        relevance_summary=evaluation_response,
+                    )
+                yield evaluation_response
+
+        else:
+            yield packet
+
+    # Saving Gen AI answer and responding with message info
+    gen_ai_response_message = create_new_chat_message(
+        chat_session_id=chat_session.id,
+        parent_message=new_user_message,
+        prompt_id=query_req.prompt_id,
+        message=answer.llm_answer,
+        token_count=len(llm_tokenizer.encode(answer.llm_answer)),
+        message_type=MessageType.ASSISTANT,
+        error=None,
+        reference_docs=reference_db_search_docs,
+        db_session=db_session,
+        commit=True,
+    )
+
+    msg_detail_response = translate_db_message_to_chat_message_detail(
+        gen_ai_response_message
+    )
+    yield msg_detail_response
+
+
+@log_generator_function_time()
+def stream_search_answer(
+    query_req: DirectQARequest,
+    user: User | None,
+    max_document_tokens: int | None,
+    max_history_tokens: int | None,
+) -> Iterator[str]:
+    with get_session_context_manager() as session:
+        objects = stream_answer_objects(
+            query_req=query_req,
+            user=user,
+            max_document_tokens=max_document_tokens,
+            max_history_tokens=max_history_tokens,
+            db_session=session,
+        )
+        for obj in objects:
+            yield get_json_line(obj.model_dump())
+
+
+def get_search_answer(
+    query_req: DirectQARequest,
+    user: User | None,
+    max_document_tokens: int | None,
+    max_history_tokens: int | None,
+    db_session: Session,
+    answer_generation_timeout: int = QA_TIMEOUT,
+    enable_reflexion: bool = False,
+    bypass_acl: bool = False,
+    use_citations: bool = False,
+    danswerbot_flow: bool = False,
+    retrieval_metrics_callback: (
+        Callable[[RetrievalMetricsContainer], None] | None
+    ) = None,
+    rerank_metrics_callback: Callable[[RerankMetricsContainer], None] | None = None,
+) -> OneShotQAResponse:
+    """Collects the streamed one shot answer responses into a single object"""
+    qa_response = OneShotQAResponse()
+
+    results = stream_answer_objects(
+        query_req=query_req,
+        user=user,
+        max_document_tokens=max_document_tokens,
+        max_history_tokens=max_history_tokens,
+        db_session=db_session,
+        bypass_acl=bypass_acl,
+        use_citations=use_citations,
+        danswerbot_flow=danswerbot_flow,
+        timeout=answer_generation_timeout,
+        retrieval_metrics_callback=retrieval_metrics_callback,
+        rerank_metrics_callback=rerank_metrics_callback,
+    )
+
+    answer = ""
+    for packet in results:
+        if isinstance(packet, QueryRephrase):
+            qa_response.rephrase = packet.rephrased_query
+        if isinstance(packet, DanswerAnswerPiece) and packet.answer_piece:
+            answer += packet.answer_piece
+        elif isinstance(packet, QADocsResponse):
+            qa_response.docs = packet
+        elif isinstance(packet, LLMRelevanceFilterResponse):
+            qa_response.llm_selected_doc_indices = packet.llm_selected_doc_indices
+        elif isinstance(packet, DanswerQuotes):
+            qa_response.quotes = packet
+        elif isinstance(packet, CitationInfo):
+            if qa_response.citations:
+                qa_response.citations.append(packet)
+            else:
+                qa_response.citations = [packet]
+        elif isinstance(packet, DanswerContexts):
+            qa_response.contexts = packet
+        elif isinstance(packet, StreamingError):
+            qa_response.error_msg = packet.error
+        elif isinstance(packet, ChatMessageDetail):
+            qa_response.chat_message_id = packet.message_id
+
+    if answer:
+        qa_response.answer = answer
+
+    if enable_reflexion:
+        # Because follow up messages are explicitly tagged, we don't need to verify the answer
+        if len(query_req.messages) == 1:
+            first_query = query_req.messages[0].message
+            qa_response.answer_valid = get_answer_validity(first_query, answer)
+        else:
+            qa_response.answer_valid = True
+
+    if use_citations and qa_response.answer and qa_response.citations:
+        # Reorganize citation nums to be in the same order as the answer
+        qa_response.answer, qa_response.citations = reorganize_citations(
+            qa_response.answer, qa_response.citations
+        )
+
+    return qa_response

backend/danswer/one_shot_answer/models.py

@@ -0,0 +1,114 @@
+from typing import Any
+
+from pydantic import BaseModel
+from pydantic import Field
+from pydantic import model_validator
+
+from danswer.chat.models import CitationInfo
+from danswer.chat.models import DanswerContexts
+from danswer.chat.models import DanswerQuotes
+from danswer.chat.models import QADocsResponse
+from danswer.configs.constants import MessageType
+from danswer.context.search.enums import LLMEvaluationType
+from danswer.context.search.enums import RecencyBiasSetting
+from danswer.context.search.enums import SearchType
+from danswer.context.search.models import ChunkContext
+from danswer.context.search.models import RerankingDetails
+from danswer.context.search.models import RetrievalDetails
+
+
+class QueryRephrase(BaseModel):
+    rephrased_query: str
+
+
+class ThreadMessage(BaseModel):
+    message: str
+    sender: str | None = None
+    role: MessageType = MessageType.USER
+
+
+class PromptConfig(BaseModel):
+    name: str
+    description: str = ""
+    system_prompt: str
+    task_prompt: str = ""
+    include_citations: bool = True
+    datetime_aware: bool = True
+
+
+class ToolConfig(BaseModel):
+    id: int
+
+
+class PersonaConfig(BaseModel):
+    name: str
+    description: str
+    search_type: SearchType = SearchType.SEMANTIC
+    num_chunks: float | None = None
+    llm_relevance_filter: bool = False
+    llm_filter_extraction: bool = False
+    recency_bias: RecencyBiasSetting = RecencyBiasSetting.AUTO
+    llm_model_provider_override: str | None = None
+    llm_model_version_override: str | None = None
+
+    prompts: list[PromptConfig] = Field(default_factory=list)
+    prompt_ids: list[int] = Field(default_factory=list)
+
+    document_set_ids: list[int] = Field(default_factory=list)
+    tools: list[ToolConfig] = Field(default_factory=list)
+    tool_ids: list[int] = Field(default_factory=list)
+    custom_tools_openapi: list[dict[str, Any]] = Field(default_factory=list)
+
+
+class DirectQARequest(ChunkContext):
+    persona_config: PersonaConfig | None = None
+    persona_id: int | None = None
+
+    messages: list[ThreadMessage]
+    prompt_id: int | None = None
+    multilingual_query_expansion: list[str] | None = None
+    retrieval_options: RetrievalDetails = Field(default_factory=RetrievalDetails)
+    rerank_settings: RerankingDetails | None = None
+    evaluation_type: LLMEvaluationType = LLMEvaluationType.UNSPECIFIED
+
+    chain_of_thought: bool = False
+    return_contexts: bool = False
+
+    # allows the caller to specify the exact search query they want to use
+    # can be used if the message sent to the LLM / query should not be the same
+    # will also disable Thread-based Rewording if specified
+    query_override: str | None = None
+
+    # If True, skips generative an AI response to the search query
+    skip_gen_ai_answer_generation: bool = False
+
+    @model_validator(mode="after")
+    def check_persona_fields(self) -> "DirectQARequest":
+        if (self.persona_config is None) == (self.persona_id is None):
+            raise ValueError("Exactly one of persona_config or persona_id must be set")
+        return self
+
+    @model_validator(mode="after")
+    def check_chain_of_thought_and_prompt_id(self) -> "DirectQARequest":
+        if self.chain_of_thought and self.prompt_id is not None:
+            raise ValueError(
+                "If chain_of_thought is True, prompt_id must be None"
+                "The chain of thought prompt is only for question "
+                "answering and does not accept customizing."
+            )
+
+        return self
+
+
+class OneShotQAResponse(BaseModel):
+    # This is built piece by piece, any of these can be None as the flow could break
+    answer: str | None = None
+    rephrase: str | None = None
+    quotes: DanswerQuotes | None = None
+    citations: list[CitationInfo] | None = None
+    docs: QADocsResponse | None = None
+    llm_selected_doc_indices: list[int] | None = None
+    error_msg: str | None = None
+    answer_valid: bool = True  # Reflexion result, default True if Reflexion not run
+    chat_message_id: int | None = None
+    contexts: DanswerContexts | None = None

backend/danswer/one_shot_answer/qa_utils.py

@@ -0,0 +1,81 @@
+from collections.abc import Generator
+
+from danswer.configs.constants import MessageType
+from danswer.natural_language_processing.utils import BaseTokenizer
+from danswer.one_shot_answer.models import ThreadMessage
+from danswer.utils.logger import setup_logger
+
+logger = setup_logger()
+
+
+def simulate_streaming_response(model_out: str) -> Generator[str, None, None]:
+    """Mock streaming by generating the passed in model output, character by character"""
+    for token in model_out:
+        yield token
+
+
+def combine_message_thread(
+    messages: list[ThreadMessage],
+    max_tokens: int | None,
+    llm_tokenizer: BaseTokenizer,
+) -> str:
+    """Used to create a single combined message context from threads"""
+    if not messages:
+        return ""
+
+    message_strs: list[str] = []
+    total_token_count = 0
+
+    for message in reversed(messages):
+        if message.role == MessageType.USER:
+            role_str = message.role.value.upper()
+            if message.sender:
+                role_str += " " + message.sender
+            else:
+                # Since other messages might have the user identifying information
+                # better to use Unknown for symmetry
+                role_str += " Unknown"
+        else:
+            role_str = message.role.value.upper()
+
+        msg_str = f"{role_str}:\n{message.message}"
+        message_token_count = len(llm_tokenizer.encode(msg_str))
+
+        if (
+            max_tokens is not None
+            and total_token_count + message_token_count > max_tokens
+        ):
+            break
+
+        message_strs.insert(0, msg_str)
+        total_token_count += message_token_count
+
+    return "\n\n".join(message_strs)
+
+
+def slackify_message(message: ThreadMessage) -> str:
+    if message.role != MessageType.USER:
+        return message.message
+
+    return f"{message.sender or 'Unknown User'} said in Slack:\n{message.message}"
+
+
+def slackify_message_thread(messages: list[ThreadMessage]) -> str:
+    if not messages:
+        return ""
+
+    message_strs: list[str] = []
+    for message in messages:
+        if message.role == MessageType.USER:
+            message_text = (
+                f"{message.sender or 'Unknown User'} said in Slack:\n{message.message}"
+            )
+        elif message.role == MessageType.ASSISTANT:
+            message_text = f"DanswerBot said in Slack:\n{message.message}"
+        else:
+            message_text = (
+                f"{message.role.value.upper()} said in Slack:\n{message.message}"
+            )
+        message_strs.append(message_text)
+
+    return "\n\n".join(message_strs)

backend/danswer/prompts/prompt_utils.py

@@ -5,11 +5,11 @@ from typing import cast
 from langchain_core.messages import BaseMessage
 
 from danswer.chat.models import LlmDoc
-from danswer.chat.models import PromptConfig
 from danswer.configs.chat_configs import LANGUAGE_HINT
 from danswer.configs.constants import DocumentSource
 from danswer.context.search.models import InferenceChunk
 from danswer.db.models import Prompt
+from danswer.llm.answering.models import PromptConfig
 from danswer.prompts.chat_prompts import ADDITIONAL_INFO
 from danswer.prompts.chat_prompts import CITATION_REMINDER
 from danswer.prompts.constants import CODE_BLOCK_PAT

backend/danswer/redis/redis_connector_credential_pair.py

@@ -10,7 +10,6 @@ from sqlalchemy.orm import Session
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
 from danswer.db.document import (
     construct_document_select_for_connector_credential_pair_by_needs_sync,
@@ -106,7 +105,7 @@ class RedisConnectorCredentialPair(RedisObjectHelper):
 
             # Priority on sync's triggered by new indexing should be medium
             result = celery_app.send_task(
-                DanswerCeleryTask.VESPA_METADATA_SYNC_TASK,
+                "vespa_metadata_sync_task",
                 kwargs=dict(document_id=doc.id, tenant_id=tenant_id),
                 queue=DanswerCeleryQueues.VESPA_METADATA_SYNC,
                 task_id=custom_task_id,

backend/danswer/redis/redis_connector_delete.py

@@ -12,7 +12,6 @@ from sqlalchemy.orm import Session
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
 from danswer.db.document import construct_document_select_for_connector_credential_pair
 from danswer.db.models import Document as DbDocument
@@ -115,7 +114,7 @@ class RedisConnectorDelete:
 
             # Priority on sync's triggered by new indexing should be medium
             result = celery_app.send_task(
-                DanswerCeleryTask.DOCUMENT_BY_CC_PAIR_CLEANUP_TASK,
+                "document_by_cc_pair_cleanup_task",
                 kwargs=dict(
                     document_id=doc.id,
                     connector_id=cc_pair.connector_id,

backend/danswer/redis/redis_connector_doc_perm_sync.py

@@ -12,7 +12,6 @@ from danswer.access.models import DocExternalAccess
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 
 
 class RedisConnectorPermissionSyncPayload(BaseModel):
@@ -133,8 +132,6 @@ class RedisConnectorPermissionSync:
         lock: RedisLock | None,
         new_permissions: list[DocExternalAccess],
         source_string: str,
-        connector_id: int,
-        credential_id: int,
     ) -> int | None:
         last_lock_time = time.monotonic()
         async_results = []
@@ -152,13 +149,11 @@ class RedisConnectorPermissionSync:
             self.redis.sadd(self.taskset_key, custom_task_id)
 
             result = celery_app.send_task(
-                DanswerCeleryTask.UPDATE_EXTERNAL_DOCUMENT_PERMISSIONS_TASK,
+                "update_external_document_permissions_task",
                 kwargs=dict(
                     tenant_id=self.tenant_id,
                     serialized_doc_external_access=doc_perm.to_dict(),
                     source_string=source_string,
-                    connector_id=connector_id,
-                    credential_id=credential_id,
                 ),
                 queue=DanswerCeleryQueues.DOC_PERMISSIONS_UPSERT,
                 task_id=custom_task_id,

backend/danswer/redis/redis_connector_prune.py

@@ -10,7 +10,6 @@ from sqlalchemy.orm import Session
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.db.connector_credential_pair import get_connector_credential_pair_from_id
 
 
@@ -135,7 +134,7 @@ class RedisConnectorPrune:
 
             # Priority on sync's triggered by new indexing should be medium
             result = celery_app.send_task(
-                DanswerCeleryTask.DOCUMENT_BY_CC_PAIR_CLEANUP_TASK,
+                "document_by_cc_pair_cleanup_task",
                 kwargs=dict(
                     document_id=doc_id,
                     connector_id=cc_pair.connector_id,

backend/danswer/redis/redis_document_set.py

@@ -11,7 +11,6 @@ from sqlalchemy.orm import Session
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.db.document_set import construct_document_select_by_docset
 from danswer.redis.redis_object_helper import RedisObjectHelper
 
@@ -77,7 +76,7 @@ class RedisDocumentSet(RedisObjectHelper):
             redis_client.sadd(self.taskset_key, custom_task_id)
 
             result = celery_app.send_task(
-                DanswerCeleryTask.VESPA_METADATA_SYNC_TASK,
+                "vespa_metadata_sync_task",
                 kwargs=dict(document_id=doc.id, tenant_id=tenant_id),
                 queue=DanswerCeleryQueues.VESPA_METADATA_SYNC,
                 task_id=custom_task_id,

backend/danswer/redis/redis_usergroup.py

@@ -11,7 +11,6 @@ from sqlalchemy.orm import Session
 from danswer.configs.constants import CELERY_VESPA_SYNC_BEAT_LOCK_TIMEOUT
 from danswer.configs.constants import DanswerCeleryPriority
 from danswer.configs.constants import DanswerCeleryQueues
-from danswer.configs.constants import DanswerCeleryTask
 from danswer.redis.redis_object_helper import RedisObjectHelper
 from danswer.utils.variable_functionality import fetch_versioned_implementation
 from danswer.utils.variable_functionality import global_version
@@ -90,7 +89,7 @@ class RedisUserGroup(RedisObjectHelper):
             redis_client.sadd(self.taskset_key, custom_task_id)
 
             result = celery_app.send_task(
-                DanswerCeleryTask.VESPA_METADATA_SYNC_TASK,
+                "vespa_metadata_sync_task",
                 kwargs=dict(document_id=doc.id, tenant_id=tenant_id),
                 queue=DanswerCeleryQueues.VESPA_METADATA_SYNC,
                 task_id=custom_task_id,