Skip to content

External Connections Setup

Projects (courses) can bring their own infrastructure instead of using the shared defaults: an external S3/MinIO bucket, an external PostgreSQL (documents + pgvector embeddings), an external Qdrant, and/or their own embedding provider. Routing is decided per project at runtime from the project_external_connections table on the host database — there are no global environment switches.

This guide covers operator setup:

  1. Prerequisites
  2. Provisioning an external Postgres (documents + pgvector)
  3. Registering a project's connections with the CLI
  4. Keeping existing external stores up to date
  5. Verifying and troubleshooting

Related pages:

Prerequisites

  • ENCRYPTION_MASTER_KEY must be set to the same value for the frontend, backend, and ingest worker — configs are stored AES-256-GCM encrypted, written by the frontend and decrypted by all three. The dev bootstrap (infra/scripts/start-dev.sh) generates one and syncs it into the app-local .env files automatically; for other deployments set it in your environment (see the root .env.template for the generation one-liner).
  • A super-admin account on the frontend. Connection CRUD is restricted to super admins (SUPER_ADMIN_EMAILS / src/utils/superAdmins.ts).
  • The host database schema must include the external-connections tables. Fresh setups get them from infra/db/init-schema.sql (applied by the setup scripts when run with --create-schema, or as part of a --clean/--wipe_data reset); existing databases created before this feature need to be recreated from it (dev: start-dev.sh --clean). The bundled postgres-illinois-chat service runs the pgvector-enabled pgvector/pgvector:pg17 image, which the schema requires.

Provisioning an external Postgres

When a project sets a database connection, its documents tables — and, if the project has no Qdrant override, its pgvector embeddings — live in that external database. The target database must be provisioned before registering the connection:

psql "postgresql://user:password@db-host:5432/dbname" \
  -v ON_ERROR_STOP=1 \
  -f infra/db/provision_external_pgvector_store.sql

Pooled providers: run this provisioning step against a direct or session-mode connection (e.g. Supabase port 5432) — it executes DDL and CREATE EXTENSION, which belong on a real session. The transaction-pooler URI recommended below is for the registered runtime connection_uri only.

The script is idempotent and project-agnostic. It creates only the ingest + pgvector core the pipeline needs on the external side:

  • documents, doc_groups, documents_doc_groups, documents_in_progress, documents_failed
  • embeddings (vector(4096)) with its GIN filters and the embeddings_embedding_1536_hnsw_idx HNSW ANN index
  • The ingest RPCs (add_document_to_group, add_document_to_group_url, remove_document_from_group) and doc-count/updated-at triggers

Requirements: PostgreSQL ≥ 15 with the pgvector extension available (≥ 0.7 for subvector). Managed providers (RDS, Supabase, …) generally ship it; the script runs CREATE EXTENSION IF NOT EXISTS vector itself.

Indexing an already-populated table: the HNSW index in the script is instant on an empty database. If you are provisioning a database that already holds millions of embeddings, read the notes inside infra/db/provision_external_pgvector_store.sql about maintenance_work_mem and serial (non-parallel) CREATE INDEX CONCURRENTLY builds first — getting this wrong turns a ~1 hour build into days.

Registering connections with the CLI

infra/scripts/add_external_project_script/external_connections_cli.py populates the host database through the frontend's authenticated API (the frontend is the sole writer and handles encryption; the CLI never sees the master key).

One-time setup:

cd infra/scripts/add_external_project_script
cp .external.env.template .external.env   # gitignored — never commit it

Its one dependency is python-dotenv; install it once with pip install -r requirements.txt (a virtual environment is recommended).

Fill in .external.env:

  • EXT_CONN_BASE_URL — the running frontend (default http://localhost:3000)
  • ACCESS_TOKEN — your super-admin JWT: log into the frontend, then copy the access_token cookie (DevTools → Application → Cookies)
  • EXT_PROJECT_NAME — default project name for all commands
  • The EXT_* block for each connection kind you plan to register (see the template's comments; only the kinds you use need filling)

Then:

# Probe a config without persisting anything (recommended first step):
python external_connections_cli.py test database

# Register the external documents/pgvector database for the project:
python external_connections_cli.py upsert database

# Other kinds work the same way (config built from .external.env):
python external_connections_cli.py upsert s3
python external_connections_cli.py upsert qdrant
python external_connections_cli.py upsert embedding

# Inspect what is stored (secrets come back masked):
python external_connections_cli.py get

# Disable / re-enable a project's overrides without deleting them:
python external_connections_cli.py set-active --active false

# Remove one kind (or the whole row when --kind is omitted):
python external_connections_cli.py delete --kind qdrant

Every command also accepts an explicit project name, a literal JSON config, or @path/to/config.json — run with --help for details.

You can also probe the config already stored for a project (decrypted server-side, never echoed back):

python external_connections_cli.py test database --stored

Supabase / pooled Postgres

For the registered runtime connection_uri, use Supabase's transaction pooler (port 6543):

postgresql://postgres.<ref>:password@<region>.pooler.supabase.com:6543/postgres

Why: the session-mode pooler (same host, port 5432) pins one database session per client connection and caps out around 15 sessions. Since every service opens its connections per request, concurrent traffic can exhaust that, producing EMAXCONNSESSION / connect timeouts. Direct connections (db.<ref>.supabase.co) bypass the pooler entirely (IPv6-only, low max_connections). Transaction mode multiplexes idle clients and avoids both problems, which is why it is the required choice rather than a preference.

The app is fully transaction-mode compatible: the frontend opens its per-request client with prepare: false (no named prepared statements) and scopes its pgvector tuning with SET LOCAL inside explicit transactions; the backend and worker use a NullPool and their psycopg2 driver needs no changes.

Session-mode and direct Supabase URIs are still accepted — the test probe and upsert respond with a warning rather than rejecting them. The URI is stored verbatim; no automatic port rewriting is done.

How routing behaves after registration

  • Vector engine: a project with an active qdrant config is served by Qdrant (via the Python backend); every other project uses pgvector — on the host Postgres, or on the project's external Postgres when a database config is set.
  • S3: uploads, downloads, and presigned URLs use the project's s3 config when present, otherwise the shared default bucket.
  • Embeddings: ingest and retrieval use the project's embedding config (provider must be in ALLOWED_EMBEDDING_PROVIDERS, default openai,ollama); otherwise the environment default.
  • Config changes take effect on the next request in every service. Nothing is cached: each request reads the project's row and builds the connections it needs, so there is no invalidation step and no restart.

Keeping existing external stores up to date

npm run db:migrate only touches the host database, and provision_external_pgvector_store.sql only runs when a store is first provisioned. Schema changes to the document/ingest tables therefore never reach an already-provisioned external Postgres on their own.

infra/db/external-migrations/ holds an ordered set of idempotent SQL scripts for exactly that. Replay the whole set against each external store after pulling a release that changed document schema:

infra/db/external-migrations/apply.sh "postgresql://user:password@db-host:5432/dbname"

Every script is safe to re-run, so there is nothing to track per store — a store that is already current is left unchanged. As with provisioning, use a direct or session-mode connection (not the transaction pooler), with a role that owns the document tables.

Freshly provisioned stores do not need this: the provisioning script is kept in sync and already includes every change.

Contributors: the convention for adding to that directory — and the rule that provision_external_pgvector_store.sql moves with it — is documented in infra/db/external-migrations/README.md.

Verifying and troubleshooting

  • test probes run server-side with SSRF protections — private/loopback addresses are rejected by design. A network error code from a probe usually means the target is not publicly reachable from the frontend.
  • 401 from the CLI: the ACCESS_TOKEN cookie value expired — grab a fresh one from the browser.
  • 403: the account is not in the super-admin allowlist.
  • Decryption warnings in backend/worker logs mean ENCRYPTION_MASTER_KEY differs between services; per-project overrides silently fall back to the shared defaults until the keys match.
  • Ingest writes fail against an external Postgres if it was not provisioned — the worker expects the RPCs and tables from provision_external_pgvector_store.sql to exist.
  • Document-group counts that look wrong only for projects on an external Postgres usually mean the store is behind on external migrations; replaying them repairs the drift as well as preventing it.
  • A config that appears not to take effect is not a caching problem: nothing is cached, so the next request already reads the new row. Check is_active, then the ENCRYPTION_MASTER_KEY match above.
  • Every write is recorded in project_connection_audit_log (actor, action, changed field names — never values) on the host database.