Operate

PostgreSQL and S3

Nipa ships two server binaries from the same module:

BinaryEntrypointMetadataChunks
nipad (free)cmd/nipadSQLite (DATABASE_DSN, default nipa.db)local (CHUNK_STORAGE=local, CHUNK_STORAGE_DIR)
nipad-ee (enterprise)ee/cmd/nipadPostgreSQL (DATABASE_DSN)S3 only (CHUNK_STORAGE=s3, empty defaults to s3)

The free binary links no ee/ package; internal/ must never import ee/. Both binaries compose repositories into the shared internal/serverapp registry and call serverapp.Run, which owns the h2c/HTTP/gRPC/SPA lifecycle. CHUNK_STORAGE=s3 in the free binary fails with a hint to use nipad-ee.

Layout

ee/
  db/                        postgres opener + migration runner
    migrations/postgres/     single consolidated schema + embed.FS
  nodelease/                 per-instance snowflake node id leases
  repository/postgres/       13 repositories (the server repository surface)
    queries/                 sqlc inputs
    sqlc/                    generated code (excluded from Sonar)
  storage/s3/                S3-compatible ChunkStore (+ DirectTransferStore)
  cmd/nipad/                 enterprise server binary
  cmd/migrate/               postgres migrate CLI
  e2e/                       testcontainers smoke suite

Schema conventions (ee/db/migrations/postgres)

  • Snowflake records keep their app-generated IDs: BIGINT PRIMARY KEY, no identity. Surrogate row ids (tree_nodes, files, chunks, pbac_rules, project_paths_permissions, refresh_tokens) use BIGINT GENERATED BY DEFAULT AS IDENTITY.
  • Every integer column is BIGINT (not INTEGER). This makes the generated Go types identical to the sqlite engine (int64 / sql.NullInt64); nullable int4 would otherwise become sql.NullInt32.
  • Hashes and payloads are BYTEA, timestamps TIMESTAMPTZ (default now()), booleans BOOLEAN, and the partial unique indexes (branches, file_locks, webhook_deliveries) mirror the sqlite predicates.
  • The migration seeds the super admin, the default org/project/branch and the owner membership, matching db/migrations/sqlite.
  • There is one consolidated 000001_init_schema (no postgres deployments exist); future changes add make migrate-create pairs.

Snowflake node ids (multi-pod)

App-generated ids embed an 8-bit node id (0–255) in internal/snow; two instances sharing a node id would generate colliding ids. nipad-ee resolves the node id as follows:

  • SNOWFLAKE_NODE_ID set (env, config.yaml or .env): that static id is used as-is.
  • unset: the instance claims a free or expired id in snowflake_node_leases (ee/nodelease) and renews the lease every SNOWFLAKE_LEASE_TTL_SECONDS / 3 (default TTL 60s) until shutdown, when the row is released. A crashed instance’s id becomes reclaimable once the TTL expires; if a lease cannot be renewed before it expires the process exits and re-claims on restart instead of risking collisions. Startup fails when all 256 ids are held by live instances.
  • The free nipad (SQLite, local chunks) is single-instance by design and keeps using SNOWFLAKE_NODE_ID (default 0).

All lease expiry math runs on the database clock (now()), so instance clock skew does not matter.

Queries and sqlc

make sqlc reads ee/db/migrations/postgres + ee/repository/postgres/queries and writes ee/repository/postgres/sqlc (package postgres). The generated models are byte-for-byte equivalent to the sqlite ones, so repository ports are mechanical. Dialect notes:

  • Use sqlc.arg('name') with quoted names: an unquoted keyword such as sqlc.arg(limit) is a parse error on the postgres engine.
  • Cast nullable filters: sqlc.narg('resolved')::boolean, sqlc.narg('last_id')::bigint.
  • LIMIT/OFFSET parameters need an explicit ::bigint (otherwise int4 is inferred); a cast directly after sqlc.arg is rejected, so write LIMIT sqlc.arg('limit')::bigint — quoting the name first is required.
  • INSERT OR IGNORE becomes ON CONFLICT DO NOTHING; excluded.* upserts and RETURNING carry over unchanged.
  • FileLockGet uses branch_id IS NOT DISTINCT FROM sqlc.narg('branch_id') to reproduce sqlite’s IS NULL semantics.

Repository layer

ee/repository/postgres implements the same unexported usecase interfaces as internal/repository/sqlite (structural typing; no interface changes needed). common.go maps errors: sql.ErrNoRows → 404, postgres 23505 → 409, anything else → 500, always preserving Cause so errors.Is(err, sql.ErrNoRows) still matches. The only intentional behavior difference is MergeRequestCreate: it allocates the per-project number while holding SELECT ... FOR UPDATE on the project row (the sqlite single writer made the old MAX(number)+1 race invisible).

Migrations

  • nipad-ee applies pending migrations at startup (ee/db.MigrateUp).
  • ee/cmd/migrate is the standalone CLI; make migrate-up-postgres DSN=... / make migrate-down-postgres DSN=... wrap it. The free make migrate-up/migrate-down remain sqlite-only.

Testing

  • Repository tests live in ee/repository/postgres and are testify suites over testcontainers: one postgres:17 container per package (TestMain), a fresh schema per test (search_path + pgxmigrate), dropped on cleanup. They assert against real FKs, so fixtures seed real trees/commits.
  • ee/db has an env-agnostic MigrateSuite; ee/e2e boots postgres + minio and runs the real client through login → clone → push (S3 presigned direct upload) → clone → tag. A second e2e test leases node ids for two in-process instances against the shared postgres and pushes through both.
  • ee/nodelease has its own testcontainers suite (fresh schema per test) covering distinct ids, expired-lease steal, exhaustion, renew, release and the lost-lease callback.
  • Overrides for service-container CI: NIPA_TEST_POSTGRES_DSN, NIPA_TEST_S3_ENDPOINT, NIPA_TEST_S3_IMAGE (default docker.io/pgsty/minio:latest), NIPA_TEST_REDIS_ADDR (rate limit store, default a redis:7 container).
  • make test-ee runs the enterprise packages; plain go test ./... runs everything when Docker is available.