Deployment and environment

Running locally (development)

Prerequisites

• Java 21+
• Maven 3.9+
• Node.js 20+ and npm
• PostgreSQL 16 with the pgvector extension, locally or via Docker. docker-compose.yml uses the pgvector/pgvector:pg16 image; a PostgreSQL 16 without the extension fails creating vector_store at startup

1. Start the database

docker compose up -d db

This starts only the PostgreSQL container, on port 5432.

2. Run the backend

cd backend

export DB_HOST=localhost
export DB_PORT=5432
export DB_NAME=flow-iagentic
export DB_USERNAME=flow-iagentic
export DB_PASSWORD=flow-iagentic
export JWT_SECRET=dev-secret-key-min-256-bits-for-hmac-sha256-algorithm

mvn install -DskipTests
mvn spring-boot:run -pl core -Dspring-boot.run.profiles=dev -Dcheckstyle.skip=true

The backend comes up at http://localhost:8080.

Two traps in this step — mvn package instead of mvn install, and the container that does not reload by itself — are covered under Local environment diagnostics, at the end of this section.

The third artifact: the runner agent

The same mvn install also produces backend/runner-agent/target/flowia-runner-agent.jar. It is neither the application nor RobotRunner.jar: it is the long-lived agent installed on a customer's machine, which fetches work from the platform and spawns the sandbox there.

The agent must not depend on core

runner-agent runs on a machine the platform does not administer. A dependency on core would carry the per-tenant datasource routing, the Flowable engine and the database driver into the customer's network — and it would compile without breaking anything. AgentCarriesNoCoreTest reads the resolved classpath and fails when that happens; it is what makes the violation surface in the build rather than on a customer's server.

The loop closes end to end: the platform assigns, the agent collects from /api/internal/runners/work, downloads both JARs verifying their SHA-256, spawns the child and reports the outcome back. The details are in specs/automation/robot-runners.md.

The runner has to be registered on the screen before the agent will start, by pasting the public key --init prints. The commands and the screen walkthrough are in Robots → Registering a runner.

3. Run the frontend

cd frontend
npm install
npm run dev

The frontend comes up at http://localhost:3000 (vite.config.ts → server.port). Serving it on another port means adding that port to app.cors.allowed-origins, or every call answers Invalid CORS request.

Local environment diagnostics

install is not optional, and package will not do

The reactor has three modules: core is the application, robot-runner is the sandbox the robots run in, and runner-agent is the agent installed on a customer's machine. core depends on robot-runner and resolves that dependency through the local ~/.m2 repository — not through the sibling's target/ directory. runner-agent depends on neither, on purpose.

mvn package builds both and publishes nothing to ~/.m2, so the cache keeps the previous robot-runner. mvn install is what refreshes it. The failure does not surface in the build: it surfaces at the next boot, as a compilation error inside the container, pointing at code you were certain you had fixed.

If you use the Docker environment, the host's ~/.m2 is mounted into the container (docker-compose.dev.yml), so the cache is shared — and so is the problem.

You ask for the container restart, the compiler does not

docker-compose.dev.yml mounts ./backend/core/target into the container, and that mount is the hot-reload mechanism: you compile on the host and the container sees the new .class files at once.

It does not reload on its own. When the build is finished, ask for the restart:

touch backend/core/target/classes/.reloadtrigger

DevTools then applies everything that changed since the last restart in one go (~1-2s, without restarting the JVM). The same applies to a build made inside the container (docker compose exec app mvn compile -q -pl core).

What holds the automatic restart back is spring.devtools.restart.trigger-file: .reloadtrigger, in application-dev.yml — the dev profile only, never prod or staging. SPRING_DEVTOOLS_RESTART_ENABLED=false would not do: spring-boot:run installs the RestartInitializer and DevTools restarts regardless, saying so in the log. The trigger file, in contrast, is read later, by the autoconfiguration — the watcher stays up and simply stops treating a new .class as a reason to restart.

Without the trigger-file, any build restarts the container — including a single-class mvn test, and including mid-build, while the classes are still half-written. The symptoms do not resemble the cause: NoClassDefFoundError, a @Configuration that never registers, a preflight refusing what it should accept.

If you touch the file and nothing happens, check that it is under target/classes (a mvn clean deletes it; touch recreates it) and that the container log shows Restarting due to ....

Environment variables (Backend)

Variable	Description	Default (dev)
DB_HOST	PostgreSQL host	localhost
DB_PORT	PostgreSQL port	5432
DB_NAME	Database name	flow-iagentic
DB_USERNAME	Database user	flow-iagentic
DB_PASSWORD	Database password	flow-iagentic
JWT_SECRET	Secret key used to sign JWTs	—
APP_ENCRYPTION_KEY	Encrypts third-party credentials at rest: AI provider keys, webhook signing secrets, secret tenant variables, the SSO client secret and mail settings. Losing it makes those unreadable, and it cannot be rotated once secrets are stored.	—
SECRETS_INJECT_INTO_PROCESS_VARIABLES	With false — recommended — secret variables never reach a process instance, so Flowable's runtime and history tables never hold a credential. Expressions use secrets.get(execution, 'key') instead.	true
AI_HTTP_TOOL_DEFAULT_POLICY	With deny — recommended — an agent with no host list makes no outbound call	allow
AI_STEP_CORE_SIZE	The installation's AI capacity: concurrent AI steps in steady state, on the dedicated pool where a step waits for the provider. This is the number to quote	8
AI_STEP_QUEUE_CAPACITY	AI steps that may wait for a slot before the pool refuses. Refused, a step is closed as a failure and is available for reprocessing	100
AI_STEP_PROVIDER_TIMEOUT_SECONDS	How long a step waits for the provider before giving up. A reservation's lease is twice this	120
FLOWABLE_EXECUTOR_CORE_SIZE	Concurrent Flowable jobs in steady state — timers, async continuations and the AI step's completion. Since 2026-09-09 it is not the AI ceiling	2
FLOWABLE_EXECUTOR_MAX_SIZE	Job pool ceiling. The extra thread is only created once the 100-slot queue is full, and by then jobs are already being rejected — it is not capacity	3
JWT_EXPIRATION_MS	Access token expiry (ms)	3600000 (1h)
SPRING_PROFILES_ACTIVE	Active Spring profile	dev
APP_NODE_ROLE	all, api or worker. Decides whether the node acquires engine jobs, runs cluster-wide scheduled work and applies the shared Flyway set. See "Node roles"	all
APP_NODE_AFFINITY_ENABLED	With true, each tenant has one owning node and the node only acquires jobs of the tenants it owns. See "Tenant affinity"	false
APP_NODE_AFFINITY_HTTP	Declares that the router in front hashes on X-Tenant-ID. It changes no routing — only the connection arithmetic, which the application cannot observe on its own	false
APP_DATABASE_MAX_CONNECTIONS	The database server's max_connections. 0 means "not stated" and turns the comparison off	0
APP_DATABASE_EXPECTED_TENANTS	How many active tenants the installation expects. Feeds the connection ceiling	0
APP_DATABASE_EXPECTED_PODS	How many backend pods the installation expects	1
APP_DATABASE_BUDGET_ENFORCE	With true the node refuses to boot when the ceiling passes APP_DATABASE_MAX_CONNECTIONS, instead of only warning	false
APP_DATABASE_ROLE_ENFORCE	With true the node refuses to boot when its database role is rolsuper or rolbypassrls, instead of only warning. See "The database role"	false
AI_FAIL_ON_EMPTY_RESPONSE	Fail the task when the agent returns an empty response, instead of writing a blank value	true
AI_LOG_RAW_RESPONSE_ON_EMPTY	Log the raw provider response when it arrives with no candidate, to diagnose why it was blocked	true
AI_TASK_FAILURE_RETENTION_DAYS	Days an AI task failure record is kept before the daily purge	90
FORM_PASSWORD_HISTORY_RETENTION_DAYS	Days a password typed into a form stays in the history, already encrypted, before the nightly sweep wipes it	7
SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE	The node's only pool, and therefore its whole budget: (pool + 1) × pods. It has a floor — see "The pool floor, and why it does not go lower on its own"	10
TENANT_POOL_MAX_SIZE	The ceiling one tenant may hold inside the shared pool. It is not a pool and opens no connection: it is the bulkhead that keeps one company from consuming everyone else's connections. The default comes from the node role: 7 on all/worker, 3 on api. Same floor	7
APP_STORAGE_TYPE	S3 or LOCAL. Outside the dev and test profiles the application refuses to boot on LOCAL	LOCAL (only usable in dev)
APP_STORAGE_S3_ENDPOINT	S3-compatible endpoint. Empty means AWS S3 itself	—
APP_STORAGE_S3_REGION	Bucket region	us-east-1
APP_STORAGE_S3_BUCKET	Bucket holding attachments, profile pictures and robot JARs. Required outside dev/test	flow-iagentic
APP_STORAGE_S3_ACCESS_KEY	Access key. Required outside dev/test	—
APP_STORAGE_S3_SECRET_KEY	Secret key. Required outside dev/test	—

Enterprise deployment (on-premise / self-hosted)

Flowi Agentic is designed to run inside the customer's own private cloud infrastructure (AWS VPC, Azure, local VM). Because of the BYOK (Bring Your Own Key) model the instance runs isolated and sends no telemetry or billing data to Flowi Agentic Inc. The only thing configured is the on-premise serial key that unlocks the licence.

# Build and start the cluster (standard node)
docker compose up --build -d

For the customer's Kubernetes cluster, use the Helm chart in the next section rather than Docker Compose.

A second replica works, and now it can have a role of its own

Scheduled work does not double up: robot CRONs, e-mails and cleanups take a database lease before running, so only one node runs each of them.

The three obstacles that used to fail silently are gone:

• Real time reaches both pods. What one node publishes travels through Postgres LISTEN/NOTIFY and is delivered to every pod's STOMP sessions. No new broker to install, and no session affinity required.
• A Java Delegate uploaded to one pod appears on the others within app.delegates.reconcile-interval-ms (60 s by default), with no restart.
• Rate limits belong to the installation. The login limiter and each API key's budget are counted in a shared table, so the number on the screen is the number that holds with one pod or with five.

Flyway also left the boot path: each tenant schema migrates on that tenant's first request and a leased sweep catches the rest, so boot time no longer grows with tenant count.

Runners scale on their own, and by a different knob. runner.replicaCount is a StatefulSet: each replica is a distinct runner with its own keypair on its own volume. Since a shared runner runs one execution at a time, it is that number — not backend.replicaCount — that decides how many robots run at once.

The second pod can now have a role. APP_NODE_ROLE=api serves HTTP without acquiring jobs, without running cluster scheduled work and without applying the shared Flyway set, on a smaller per-tenant ceiling; APP_NODE_ROLE=worker does the opposite and answers only health over HTTP. Without the variable a node is all, which is the behaviour it always had. See "Node roles".

What decides the pod count is the connection budget, and it multiplies per pod only: (sharedPool + 1) x pods — the parenthesis is what one pod costs, and the +1 is the connection it holds open to listen for events. Since 2026-09-09 there is no per-tenant term in that sum. The application does the arithmetic and prints it on every boot; state APP_DATABASE_MAX_CONNECTIONS and APP_DATABASE_EXPECTED_PODS and it warns when the number does not fit (APP_DATABASE_EXPECTED_TENANTS is still printed on that line, precisely to show that it does not move it). See "Connection budget".

The ReadWriteOnce volume is still the obstacle the chart detects and refuses. With S3 required it is only a cache: persistence.enabled=false takes that one out of the way.

Docker services

Service	Container	Port
Database	flow-iagentic-db	5432
Backend	flow-iagentic-app	8080

Frontend

The frontend does not run in Docker by default — run it locally with npm run dev.

docker-compose brings up no runner; the chart does

docker-compose neither brings up nor registers a runner: here a robot only executes after someone registers a runner on the screen and starts the agent on its machine — see Registering a runner is part of bringing the installation up.

Helm has been different since 2026-08-29. The chart brings up runner.replicaCount runners (1 by default), each generating its own keypair inside its own volume and enrolling itself as GLOBAL. The private key is born on the pod that signs with it and never travels. A runner scoped to a named tenant still requires someone pasting a public key on the screen.

Kubernetes deployment (Helm)

The chart lives in helm/ and installs backend + SPA + runners into the customer's cluster. It pulls no dependencies: there is no remote subchart, so the install works in a cluster with no route to the internet, as long as the images are reachable.

Requirements

Kubernetes	1.23+
Helm	3.8+
Ingress controller	any — or none, with ingress.enabled=false
PostgreSQL	16 with pgvector, embedded or external
Object storage	required — any S3-compatible endpoint, plus a bucket and a key that can read, write and delete in it

Evaluation — embedded database, your bucket

cd helm
helm install flowia ./flowia -n flowia --create-namespace --wait --timeout 20m \
  --set storage.s3.endpoint=https://s3.customer.com \
  --set storage.s3.bucket=flowia-attachments \
  --set storage.s3.accessKey=... --set storage.s3.secretKey=...
kubectl -n flowia port-forward svc/flowia-frontend 8080:80

Comes up with embedded PostgreSQL and generated secrets. The four storage flags are not optional: object storage is a requirement, and an install without them fails at render time rather than when somebody attaches a file. The first boot runs Flyway on the shared schema and on every tenant schema — it takes minutes, and the startupProbe waits up to 600s before declaring the pod dead. Do not shorten that thinking it has hung.

port-forward does not carry real time

The task inbox's live updates use a WebSocket at /ws, and what routes /ws is the Ingress — not the frontend's nginx. Without an Ingress the screen works but does not refresh itself.

Production

helm install flowia ./flowia -n flowia --create-namespace \
  -f examples/production.yaml --wait --timeout 20m

helm/examples/production.yaml is the shape, not the values: the customer's database, the customer's ingress, the customer's managed secrets.

The database is an on/off switch

# embedded (default) — pgvector StatefulSet + PVC, for evaluation
--set postgresql.enabled=true

# external — the database the customer already runs
--set postgresql.enabled=false \
--set externalDatabase.host=pg.customer.com \
--set externalDatabase.database=flowia \
--set externalDatabase.username=flowia \
--set externalDatabase.existingSecret=flowia-db

The embedded database has no backup, no failover and no pooler — it exists so an evaluation installs in one command. The external one must allow CREATE EXTENSION vector (Spring AI creates it at boot) and must let the application create schemas, because every tenant is a schema.

With externalDatabase.host missing, helm template fails immediately with the written message — rather than a CrashLoopBackOff half an hour later.

Object storage is a requirement, not an option

Every installation requires S3. storage.type defaults to S3, and storage.s3.endpoint and storage.s3.bucket have no defaults: leave either one empty and helm template fails immediately with a written message — the same way it fails without externalDatabase.host. A helm install with no values at all is not valid.

--set storage.s3.endpoint=https://s3.customer.com \
--set storage.s3.bucket=flowia-attachments \
--set storage.s3.accessKey=... --set storage.s3.secretKey=...
# or: --set secrets.existingSecret=flowia-secrets, carrying S3_ACCESS_KEY and S3_SECRET_KEY

storage.type=LOCAL is development and test only. There the volume at /datais attachment storage: one ReadWriteOnce disk with no backup, no replication, no lifecycle policy and no second reader — losing it loses every attachment in the installation. The application refuses to boot on LOCAL outside the dev and test Spring profiles, so a chart that renders LOCAL will not produce a running pod.

storage.type	The volume at /data	persistence.enabled=false
S3 (default)	is only a cache of robot JARs	allowed — becomes an emptyDir
LOCAL (dev/test)	is attachment storage	refused by the chart

A robot's JAR is written to durable storage on upload and pulled down on demand when it is missing from disk, which is why the cache can be ephemeral. persistence.enabled defaults to true so a fresh pod does not have to fetch every JAR from the bucket before its first execution; on S3, turning it off costs that latency and nothing else.

The chart does not deploy MinIO. Point storage.s3.endpoint at the object storage the customer already has, including a MinIO they run themselves, which is their licence and their decision. An empty endpoint is refused rather than treated as "AWS S3": an on-premise release that quietly uploaded the company's attachments to Amazon is worse than one that does not start.

File retention is a bucket rule, not a job of ours

Expiring an old attachment is not the platform's business: it ships no purge job for stored files and will not grow one. Expiring an old object is a bucket lifecycle rule (or your appliance's ILM), configured by whoever owns the bucket, next to the retention and legal-hold policy the business already has.

Deleting an attachment in the product deletes its object immediately — that is deletion on demand, not retention. If you do write an age-based rule, scope it so it cannot expire objects the database still references, or the attachment comes back to the user as a download error.

Migrating an installation that already runs on LOCAL

The upgrade does not move the files. They are on the PVC; neither Helm nor the backend copies anything. Nothing in this change deletes an attachment, and nothing in it copies one either. In order:

1. Provision a bucket on the object storage the customer already runs, plus a key that can read, write and delete inside it.
2. Copy the files out of the volume. They live under persistence.mountPath — /data/attachments by default — laid out as <directory>/<fileName>, which is exactly the S3 key S3StorageProvider would have written. An mc mirror or aws s3 sync of /data/attachments/ into the bucket root reproduces the layout. Copy from a pod with the volume mounted, or from a snapshot; the backend may keep running.
3. Rewrite attachments.storage_path. This is the step that is easy to forget and impossible to skip: under LOCAL the column holds an absolute path (/data/attachments/<directory>/<fileName>), under S3 it holds the key (<directory>/<fileName>). attachments is a per-tenant-schema table, so the update runs once in every tenant_<slug>, not once globally.
4. Only then upgrade, with the storage values set.

There is a shortcut for a small installation, and it is a shortcut: re-upload the attachments through the UI and let the platform write them to S3 itself. It works, it produces new rows with new keys, and it does not scale past a few dozen files.

Memory: robots do not run inside the backend container

No robot opens in the application's JVM: every execution is assigned to a registered runner, which is another machine (or another pod) running the agent. backend.resources sizes the platform alone, and the sandbox heap ceiling belongs to the agent, on the agent's machine — the chart has no key for it.

Whoever sizes memory for robots sizes the runner's machine. An installation with no runner registered spends no memory on robots — and runs none either.

Registering a runner is part of bringing the installation up

Without at least one runner registered and answering, executions stay PENDING forever, with the reason written on the row itself. The platform runs no robot of its own.

With Helm the chart already handles this: it brings up runner.replicaCount runners (1 by default) that generate their own key inside their own volume and enrol themselves as GLOBAL. Only enrolment from inside the release does that, and it can only ever create a GLOBAL runner — holding the token buys a share of the shared pool, not a tenant's robot work.

Outside Helm, and for any runner scoped to a named tenant, registration is manual, and deliberately stays so: deciding which machine receives a tenant's code is a decision for a person (specs/automation/robot-runners.md → How the chart installs runners).

The step runs once per machine, and the commands and screen walkthrough are in Robots → Registering a runner. What belongs to whoever brings the installation up is app.api.base-url: the agent probes the platform's RPC address at startup and stops, naming the address, when it cannot reach it. localhost there is the platform itself, never the runner's machine. The chart fills this in on its own — the Ingress URL when there is one, the in-cluster Service otherwise — and app.apiBaseUrl in the values overrides it.

The runner must not be the application's own machine

Installing the agent on the backend's host puts the tenant's code back on the disk holding every other tenant's configuration, credentials and attachments — exactly the exposure that executing outside the application avoids. The chart does not prevent it; the choice belongs to whoever installs.

The operating envelope: what one installation holds

Before the arithmetic, the honest summary: this platform's scalability is bounded by the database, and that is a consequence of schema-per-tenant isolation, not a defect to fix. Every tenant gets its own schema, and that is what keeps one company from seeing another's data. What the database charges, however, no longer grows with the number of companies: since 2026-09-09 there is a single pool per node, and the cost belongs to the pod.

Three limits decide the size of an installation. Only one of them is HTTP, and it is not the tight one.

Limit	What fixes it	How it shows up
Database connections	(sharedPool + 1) × pods	FATAL: sorry, too many clients already, on a user's request
AI step concurrency	AI_STEP_CORE_SIZE (8 by default), per installation	the AI step queue grows; waiting increases, and a step refused by a full queue goes to reprocessing
HTTP	pod CPU and memory	rarely the bottleneck here

Sizing by the database. With the default (sharedPool 10) every pod costs 11 connections — the 10 of the pool plus the cluster-bus listener — and that number does not move with the number of customers. How many pods fit, leaving out the headroom Postgres reserves for superusers and your own psql session:

max_connections	Typical Cloud SQL tier	pods that fit, at any number of tenants
25	db-f1-micro	2
50	db-g1-small — the managed deployment's	4
100	1 vCPU / 3.75 GB	9
200	2 vCPU / 7.5 GB	18
400	4 vCPU / 15 GB	36

The arithmetic, so any row can be checked: (10 + 1) × pods.

Until 2026-09-09 this table had a column per installation shape and a tenant count in every cell, because each tenant opened its own HikariCP pool and the sum was (sharedPool + poolPerTenant × activeTenants + 1) × pods. The per-tenant pool was removed: one pool serves everyone, the search_path is applied when a connection is borrowed and the session is reset when it is returned. The next customer stopped moving the number.

The ceiling arrives with the pod — and that is a lever

Redundancy used to cost tenants: two pods on a db-f1-micro did not sustain even one active customer, and the third customer took the redundancy off a db-g1-small. Now the only factor is the pod, and its price is fixed: another pod costs 11 connections and brings another Flowable executor with it. That is why scaling out became the way to buy job throughput — with a pool per tenant it was not, because each pod also multiplied the term for the tenants it served.

The shared pool has a floor, and the floor is not negotiable

An AI step holds two connections at its peak (the quota deduction is REQUIRES_NEW). Since every connection comes from the same pool, it needs app.flowable.executorMaxSize × 2 + 1 — 7 with the shipped executor. The chart refuses the render below that and the application warns at startup, because the failure it prevents is not too many clients: it is a tenant that quietly stops finishing AI steps, with no log and no error. The same floor applies to app.tenantPool.maxSize, which is the ceiling one tenant may hold inside the shared pool.

What the shared pool costs, measured rather than assumed:

• ~1.8x per short query, on a bench with four tenants alternating on the same physical connection (SharedPoolCheckoutCostMeasurementTest, perf profile). That is ~51 µs of extra round trip per checkout (applying the search_path and resetting the session) and ~15 µs of re-planning, because PostgreSQL re-analyses a statement when the search_path changes. At AI-step scale the difference disappears: with 1 s and 3 s model calls the sustained throughput is identical.
• The bulkhead is a ceiling, not a reservation. A pool per tenant guaranteed by construction that one tenant could not consume another's connections. What replaces it is a per-tenant ceiling inside the shared pool (app.tenantPool.maxSize), exact per node — but a ceiling, not a reservation: one tenant can wait because of another's burst.

Tenant affinity: one owning node per tenant

app.tenantAffinity.enabled (off by default) gives each tenant one owning node, the only one that acquires its jobs. Affinity's connection argument died with the per-tenant pool: there is no per-tenant term left for it to stop multiplying, and the budget is identical with it on and off. The startup line says so rather than staying silent about a benefit that no longer exists.

What it still buys, and what it asks in return:

• Job ownership and plan locality. A node acquires only its own tenants' jobs, and the database sees the same tenant through the same node.
• The router in front must hash on X-Tenant-ID. The chart writes nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x_tenant_id" when app.tenantAffinity.http is on, and refuses the render when ingressClassName is not nginx — another controller would ignore the annotation silently.
• A node that dies gives its tenants back in 30–40 s (one heartbeat plus the TTL). During that window those tenants' jobs wait; nothing is lost and nothing fails, but they are late. The details, including what each node logs, are in specs/core/tenant-node-affinity.md.
• A single-pod installation does not change at all — with one destination there is nothing for a hash to spread.

Sizing by AI. One installation runs 8 AI steps at a time (AI_STEP_CORE_SIZE), and throughput is AI pool core-pool-size × 60 ÷ latency_in_seconds. With ~20 s calls that is 24 steps per minute — for the whole installation, not per tenant. Past that the 100-slot queue grows and each step waits longer; past the queue, a step is closed as a failure and left for reprocessing rather than parked in silence. That is arithmetic: the bench with injected latency (specs/core/ai-step-concurrency.md) measured the old, 2-slot design.

What this design does not solve, and will not

Said once, so nobody discovers it in production:

• A busy tenant can make another wait. Connections no longer grow with the number of customers, but there is one pool: the per-tenant ceiling keeps one company from taking everything, and it reserves nothing for a company that is idle. Heavy concurrent writing inside a single tenant is the case this design has not measured — recorded as a known limit in specs/core/connection-pool-budget.md.
• Flowable's tables are shared. The 53 ACT_* tables live in public, filtered by tenant in the query. A tenant whose history dwarfs the others affects everyone's query times: the isolation there is of data, not of performance.
• AI throughput belongs to the provider. No configuration of ours makes a 20-second call faster. What you choose is how many run in parallel — and since 2026-09-09 each holds a thread of the AI pool, but not a database connection while it waits.
• The numbers here are worst case. They add up every pool being full at once. A real installation usually sits below them — and "usually" is not something to size against.

The recipe, in three lines

1. Count the tenants that will be active at the same time, not the ones registered.
2. Pick the tier from the table above, with headroom for your psql session and for peaks.
3. Only then decide replicas: with two pods, redo the arithmetic — and if it does not fit, split the roles before raising the tier, because an api node costs 3 connections per tenant instead of 7.

Connection budget: only the pod multiplies

There is one HikariCP pool per pod (spring.datasource.hikari.maximum-pool-size). Every tenant request comes out of it: SchemaBoundDataSource points the borrowed connection at tenant_<slug> and resets the session when it goes back. An installation's connection demand is:

(sharedPool + 1) × pods

The + 1 × pods is the cluster-bus listener (LISTEN flowia_cluster), which is how a real-time update published on one pod reaches a browser connected to another. Fixed cost per pod, not per tenant.

There is no tenant term. APP_DATABASE_EXPECTED_TENANTS still exists and still appears on the startup line, but it enters no sum: it is there so the operator sees, side by side, the customer count they expected and a budget that does not move with it.

The chart templates this, and knows how to refuse

externalDatabase:
  poolMaxSize: 10       # the ONE pool of each pod (floor: app.flowable.executorMaxSize × 2 + 1)
app:
  tenantPool:
    maxSize: 7        # ceiling ONE tenant may hold inside that pool (TENANT_POOL_MAX_SIZE)
    apiMaxSize: 3     # the same ceiling on a node with the `api` role
  database:
    maxConnections: 200   # the server's max_connections. 0 means "not told", and nothing is checked
    expectedTenants: 8    # printed at boot; it does not enter the sum
    expectedPods: 0       # 0 derives it from backend.replicaCount + backend.worker.replicaCount
    budgetEnforce: true   # refuse the `helm template` instead of refusing the connection

With budgetEnforce: true a release that does not fit fails to render, printing the sum and the ceiling, instead of installing and meeting the limit as FATAL: sorry, too many clients already under load. expectedPods: 0 is the default and the right value: the chart counts the pods it creates itself, so the number the platform is told is the number the cluster runs. Fill it in only when pods outside this release share the same database.

The application prints the same arithmetic, and can refuse too

The chart's guard catches a release before it installs. The node catches a pod configured past the chart — through backend.extraEnv, a hand-edited manifest, or a database whose tier changed under it. On every boot the backend prints one line with the node's role and the decomposition, not just the total (the log itself is written in Portuguese):

Orçamento de conexões (papel=all, pior caso): (10 compartilhado + 1 bus) = 11 por pod x 2 pod(s) = 22 conexões, contra APP_DATABASE_MAX_CONNECTIONS=50. Não há parcela por tenant: ...

The numbers come from APP_DATABASE_MAX_CONNECTIONS and APP_DATABASE_EXPECTED_PODS, which the chart fills in from app.database.*. Past the ceiling it warns loudly with the overshoot and the levers that remain; with APP_DATABASE_BUDGET_ENFORCE=true the node refuses to boot instead of warning. The default is false on purpose: refusing by default would turn one wrong number in values.yaml into total unavailability, which is worse than the problem being reported.

The number is worst case, not current consumption

It is the ceiling the shared pool may reach; spring.datasource.hikari.minimum-idle decides how many of those connections stand up while the node is idle. Reading the warning as "the database is already full" leads to resizing the wrong thing.

The two ways out, and when each is the right one

When the sum does not fit two moves remain — and the one that disappeared is the news:

Move	When it is the right one	What it costs
Raise the database tier	When the installation needs more pods, for job throughput or for redundancy	Money, and a Cloud SQL maintenance window
Fewer pods (backend.replicaCount, backend.worker.replicaCount)	When the current throughput is enough and connections are what is short	Redundancy, or job throughput — every pod carries an executor
Split the roles	No longer a connection lever. An api node and a worker cost the same: one pool each. Splitting still pays for load isolation, not for connections	—

And the move that stopped being necessary: raising the tier before an onboarding. The next customer does not move the number any more; what moves it is the decision to run another pod, and that decision is yours rather than theirs.

Shrinking the pool has a floor

externalDatabase.poolMaxSize (and app.tenantPool.maxSize) do not go below app.flowable.executorMaxSize × 2 + 1. Below that the AI steps hold and wait on each other, and the symptom is not an error: it is the tenant no longer finishing steps. Lowering the pool requires lowering FLOWABLE_EXECUTOR_MAX_SIZE with it — and then the installation runs fewer AI steps in parallel.

Two corrections worth knowing about:

• Until 2026-09-03 this formula counted sharedPool once, and the shared pool is per pod — each pod is a JVM with its own. The old version under-reported a two-pod install by a whole pool and read "fits" where the truth was "does not".
• Until 2026-09-09 it carried a poolPerTenant × activeTenants term. That left with the per-tenant pool. If you find a copy of either old formula anywhere, it is wrong. Recorded in specs/core/connection-pool-budget.md.

What the managed deployment runs today

The managed deployment runs two pods against a Cloud SQL db-g1-small (max_connections ≈ 50), with SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE=10:

pods	arithmetic	against 50
2	(10 + 1) × 2 = 22	fits
3	(10 + 1) × 3 = 33	fits
4	(10 + 1) × 4 = 44	fits, with little headroom for your own psql session
5	(10 + 1) × 5 = 55	does not fit

On any row, at any number of tenants. Until 2026-09-09 the same instance carried two active customers and ran out on the third; and on the db-f1-micro it was until 2026-09-05, the first customer already did not fit.

Changing the tier restarts the instance

gcloud sql instances patch --tier drops the open connections. Doing it with a customer already on the platform means scheduling downtime with them. That is why a tier change is a pre-onboarding item and not an incident-response one.

Required TLS and backups on the managed instance

Two other properties of the managed instance changed in the same window on 2026-09-05:

Property	Value	What it forces
settings.ipConfiguration.sslMode	ENCRYPTED_ONLY	A cleartext connection is refused by the server, not silently downgraded. Anything connecting over IP needs sslmode=require in the JDBC URL; the Cloud SQL connector negotiates TLS on its own
Automated backups	daily, with point-in-time recovery	There is a restore point that does not depend on someone remembering to run gcloud sql backups create

Point-in-time recovery restores the instance, not one customer

It returns the whole server to an instant: every tenant_<slug> schema and public, with the ACT_* tables and the discriminator tables of every tenant together. There is no "roll tenant acme back to yesterday" down this path — using it to fix one customer reverts the others with it.

What does exist for a single tenant is the pair POST /api/admin/tenants/{id}/content-export / POST /api/admin/tenants/{id}/restore, which runs pg_dump -n tenant_<slug> and restores over that schema. That is not a tenant backup: the dump covers only that schema, so running instances, tasks, jobs, history, AI agents, robots, API keys and webhooks — all of which live in public — are left out. See Multi-tenancy for which tables live where.

The pool floor, and why it does not go lower on its own

externalDatabase.poolMaxSize is the obvious lever for fitting more pods, and it has a floor that is correctness, not tuning. The AI quota deduction runs in its own transaction (REQUIRES_NEW), which suspends the job's transaction without releasing its connection and takes a second one. Every AI step therefore holds two at its peak:

floor = app.flowable.executorMaxSize × 2 + 1
      = 3 × 2 + 1
      = 7

The + 1 is the connection that has to be left for HTTP traffic. The floor applies to the smaller of externalDatabase.poolMaxSize (the pool a job draws from) and app.tenantPool.maxSize (that tenant's ceiling inside it): 20 free connections do not help a tenant whose ceiling is 3, because its steps hold and wait on each other inside the ceiling.

Below the floor the symptom is not an error — it is the tenant no longer finishing AI steps. Each job holds its first connection and waits for a second one that only frees when another job finishes, and none does. What shows up in the log is connection timeouts on flowable-task-exec-* threads, with nothing saying "the pool is too small".

The lock-up itself starts earlier than the floor suggests, and the distance is deliberate. On the rig in specs/ai/quota-transaction-boundary.md the pool stops making progress at executorMaxSize and resumes at executorMaxSize + 1 — 4 with today's executor. Between 4 and 7 nothing locks up: steps merely wait for a connection. Those three of margin cover what the rig does not model, the row lock on the tenant's single quota row, which makes concurrent deductions queue while each caller still holds its connection. The guard is written as × 2 + 1 because that is the worst case; + 1 is only where it stops being fatal.

That is why shrinking the pool alone is refused at chart render time, with the arithmetic in the message. Shrinking both together stays allowed, and is the configuration that buys another pod:

externalDatabase:
  poolMaxSize: 5       # brings a pod down to 6 connections
app:
  flowable:
    executorMaxSize: 2   # lowers the floor to 5
  tenantPool:
    maxSize: 5

What that trade gives up is the third thread, which ThreadPoolExecutor only creates once the 100-slot queue is already full — the last relief before a job is rejected rather than merely delayed. Sustained throughput does not change: it is executorCoreSize, still 2.

app.tenantPool.apiMaxSize sits outside this arithmetic. An api node acquires no jobs, so no thread on it opens the quota transaction and none ever holds two connections — that, and not extra tolerance, is what allows the pool of 3 there.

The full derivation and the measured configurations are in specs/ai/quota-transaction-boundary.md.

The database role: ordinary, never superuser

The application connects with an ordinary login role. It owns the public schema and has CREATE on the database — enough for the migrations, for provisioning's CREATE SCHEMA tenant_<slug> and for backup's pg_dump/psql — and it does not have SUPERUSER, BYPASSRLS, CREATEROLE, CREATEDB or REPLICATION.

This matters for one direct reason: a superuser role ignores schema permission and ignores Row Level Security, including FORCE ROW LEVEL SECURITY. With it, no database-level defence — present or future — works, and the whole isolation between customers rests on application code alone. SUPERUSER also enables COPY … FROM PROGRAM, which runs a command on the database host.

Check your installation with this command, substituting the name if you use a different externalDatabase.username:

SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'flow-iagentic';

Or straight from a shell:

psql "$DATABASE_URL" -tAc \
  "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'flow-iagentic'"

The right answer is flow-iagentic|f|f. Any t means the installation has no database-level barrier between customers.

To fix it, as a database superuser:

ALTER ROLE "flow-iagentic" NOSUPERUSER NOBYPASSRLS NOCREATEROLE NOCREATEDB NOREPLICATION;
GRANT CONNECT, CREATE ON DATABASE "flow-iagentic" TO "flow-iagentic";
ALTER SCHEMA public OWNER TO "flow-iagentic";

The backend checks this itself on every boot and prints the line:

Role de banco: flow-iagentic (rolsuper=false, rolbypassrls=false). Nenhuma defesa no nível do banco é ignorada por esta conexão.

When the role carries too much privilege, the same line becomes a warning and names the attribute. APP_DATABASE_ROLE_ENFORCE decides what happens: false warns and boots — the default, because on an on-premise installation the role is the customer DBA's decision, and a platform that will not start gets rolled back, taking the warning with it; true refuses to boot.

Where each default sits, and why:

Where	Default	Reason
docker-compose.yml / .dev.yml	true	An environment we control, and the fix is docker compose down -v
docker-compose.prod.yml	false	An existing volume is not re-provisioned; migrate the role, then turn it on
Helm chart (app.database.roleEnforce)	false	The chart is the on-premise delivery. helm/examples/production.yaml turns it on, after the query above answered f and f

The chart's bundled database runs the application as a superuser

postgresql.enabled=true brings up a StatefulSet whose POSTGRES_USER is the application's own role — a superuser, by construction of the image. It exists so an evaluation can start in one command, and the chart README already says it has no backup, no failover and no pooler. This is one more reason not to use it for anything real: use externalDatabase with a role provisioned as above.

An existing database volume is not re-provisioned

initdb scripts only run against an empty volume. A development environment created before this change still connects as a superuser — the backend refuses to boot and says why. docker compose down -v recreates the database from scratch. That is development data; on a database with real data, use the ALTER ROLE above instead of deleting the volume.

Node roles: which pods answer HTTP and which execute jobs

One image, two jobs, chosen by configuration. The default is one node doing both, which is what every existing installation has — the split is opt-in and changes nothing for anyone who does not ask.

Role	Serves HTTP	Executes Flowable jobs	Cluster scheduled work and robot CRONs	Migrates the shared schema	Default TENANT_POOL_MAX_SIZE
all	yes	yes	on the elected node	yes	7
api	yes	no	no	no	3
worker	/actuator only	yes	on the elected node	yes	7

The variable is APP_NODE_ROLE, and the chart writes it into both Deployments. A node that boots without it is all — exactly the behaviour it always had, which is why no existing installation changes on upgrade. A misspelled role refuses to boot rather than becoming all and serving under a role other than the configured one.

backend:
  role: api          # this Deployment only answers HTTP
  replicaCount: 2
  worker:
    enabled: true    # and this one only executes jobs
    replicaCount: 2
persistence:
  enabled: false     # four pods cannot mount one ReadWriteOnce volume

An api node still creates jobs: starting an instance, completing a task and committing a variable all write to ACT_RU_JOB. It simply never acquires them.

"Runs no scheduled work" means cluster work. What separates the two is the lease: sixteen of the nineteen scheduled methods take a lease in job_leases before running, and on an api node that claim is denied immediately, without even reaching the database. The three that take no lease keep running on every node on purpose, because what they repair or publish is that node's state: delegate reconciliation, the AI queue's saturation sampling, and the node's membership heartbeat (ClusterMembershipService). ScheduledWorkIsLeasedOrPerNodeTest holds that list of three and fails a new @Scheduled method that does neither.

"Does not migrate" is the shared set at boot. Tenant-schema Flyway stays on in an api node — it is what creates a new tenant's schema and migrates it on that tenant's first request, and turning it off would hand a new tenant an empty schema and report success.

When to split. It is not about HTTP. The throughput that runs out is AI steps: one spends seconds to minutes blocked on the model holding a thread and a tenant connection, and that is the number adding workers grows. If the bottleneck is HTTP requests, backend.replicaCount already answers it.

Who migrates. The roles that execute jobs (all and worker) run the shared schema's Flyway set at boot; api never does. That is why backend.role: api without backend.worker.enabled: true is refused at render time: nothing would execute a job and nothing would migrate, and neither failure announces itself — the work simply accumulates unacquired. The reasoning, and the alternative that was rejected (a pre-upgrade Job), are in specs/core/job-worker-nodes.md → Who migrates.

Nothing orders the two Deployments

Kubernetes starts api and worker at the same time. An api pod can come up while the migration is still running; the right behaviour is for it to refuse readiness until the schema is the one it expects, and that half is not implemented yet in the application. Until it is, on an upgrade that carries a migration, install with --wait and check the worker's rollout before sending traffic.

Secrets

Leave secrets.jwtSecret and secrets.encryptionKey empty and the chart generates them, re-reading the existing Secret on upgrade so it never rotates by accident.

Keep the APP_ENCRYPTION_KEY

It decrypts credentials already stored in the database: the SSO secret, the SMTP password, the AI provider key (ai_models, rag_config), the webhook signing secrets and every tenant variable marked secret. Losing it does not fail immediately — it fails hours later, on the first integration that tries to read its own password. The Secret carries helm.sh/resource-policy: keep so helm uninstall does not take it along, and the installation notes print the backup command.

What is encrypted in the database

Third-party credentials are not stored in clear text. The columns below are encrypted with AES-256-GCM using APP_ENCRYPTION_KEY, on write and on read, transparently to the application:

Column	What it holds
ai_models.api_key, rag_config.api_key	The customer's credential with the AI provider
webhooks.secret	The key signing each delivery's HMAC
tenant_variables.value_text	Tenant variables, including the ones marked secret
sso_connections.client_secret, e-mail settings	SSO and mail-server credentials
ACT_RU_VARIABLE.TEXT_, ACT_HI_VARINST.TEXT_ (type encryptedSecret)	A password typed into a form field of type Password

At boot the application encrypts any value still in clear text and logs how many there were; the expected number is zero.

The last row is the only one that expires: a password typed into a form leaves the runtime tables when its instance ends, and a nightly sweep wipes the historic copy after FORM_PASSWORD_HISTORY_RETENTION_DAYS days (7 by default). It is the one credential with no home in the vault — it is born mid-process and belongs to a single instance.

A request body never reaches the log

The HTTP request log records from status 400 up and never writes the body that was sent, on any route. This depends on no field being marked and no path being listed: a form submission that fails carries the typed values in its body, and one of them may be a password. What stays in the log is the method, the path, the query (with password and token already obfuscated), the headers (with Authorization and X-API-Key already obfuscated), the status and the response body — which is where the validation error explaining the failure lives.

Credentials the system only checks are not encrypted but hashed, which is stronger: user password, API key, session token and e-mail token. The difference is whether the original value has to come back: the provider key has to be sent to the provider again, a password never does.

Encryption at rest protects the database, not the application

A backup dump, a read replica or direct database access do not reveal these credentials. A compromised process of the application itself does — it holds the key and decrypts by construction.

Recommended: SECRETS_INJECT_INTO_PROCESS_VARIABLES=false

With true (the default) a secret variable is copied into the instance's variables and Flowable stores it in clear text in the runtime and history tables. With false it never enters the instance — expressions use secrets.get(execution, 'key') instead, and no credential lands in the engine's tables. See Tenant Variables.

To manage secrets externally, use secrets.existingSecret with the keys: DB_PASSWORD, JWT_SECRET, APP_ENCRYPTION_KEY, S3_ACCESS_KEY, S3_SECRET_KEY.

GEMINI_API_KEY is optional — include it to configure a bootstrap AI model; without it the platform boots and AI features fail on the call, with a 409 and a sentence saying where to register a model. Do not include it empty: an empty variable is set, not absent, and Spring AI refuses to start against one. That is how the backend used to CrashLoopBackOff while helm install reported success.

AI provider

The default is Gemini through its OpenAI-compatible API, but any compatible endpoint works — vLLM, an internal gateway, Ollama behind a proxy:

ai:
  baseUrl: http://llm-gateway.internal:8000/v1/
  chatModel: qwen2.5-72b-instruct
  embeddingModel: bge-m3
  embeddingDimensions: 1024

embeddingDimensions sizes the pgvector column: changing it after documents are indexed requires reindexing the whole knowledge base.

Cluster with no internet

cd helm
make mirror REGISTRY=registry.corp.internal/flowia TAG=1.0.0
helm install flowia ./flowia -n flowia --create-namespace -f examples/airgapped.yaml

Nothing else reaches out: no remote subchart, no init container that downloads anything.

Operating it

cd helm
make status                  # release, pods, services, volumes
make logs                    # follows the backend
make smoke                   # helm test: backend health + SPA from inside the cluster
make upgrade VALUES=my.yaml

After installing, log in as SUPER_ADMIN and paste the installation's licence under Global (Super Admin) → Licence. Verification is offline: there is no activation server and no need for outbound internet access.

Chart tests

cd helm && make test

The assertions need only helm — no cluster, no plugin, no network. They render the chart under several sets of values and check object shape, database wiring, ingress routes, storage modes, secret handling and every guard that has to fail at render time rather than in the pod.

CI runs this on every push, plus kubeconform -strict against the real Kubernetes API schemas and helm package, publishing the .tgz as a build artifact.

Flyway — database migrations

Two sets, applied at different moments:

Path	Scope	When it runs
src/main/resources/db/migration/	The public (shared) schema	At startup, and only on the roles that execute jobs (all and worker) — an api node assumes the schema is already migrated
src/main/resources/db/tenant/	Each tenant's schema	At tenant provisioning; on that tenant's first request on each node; and on the background sweep, for a tenant with no traffic

No node migrates any tenant at startup

Since 2026-08-28 boot time no longer grows with the number of tenants. The consequence for whoever deploys is that a tenant schema's migration arrives with that tenant's first request after the release, or with the sweep — not with the pod coming up. See Multi-tenancy.

The tenant path is db/tenant/, not db/migration/tenant/

The shared Flyway points at classpath:db/migration and sweeps everything nested under it. A tenant migration placed in db/migration/tenant/ would also be executed against the public schema — the exact opposite of what it is for. See Multi-tenancy for the full mechanism.

Rehearse the migration on a copy before shipping it

A migration that only creates a table needs no rehearsal. One that rewrites existing rows does: how many rows it touches — and which ones it cannot convert — only shows up against real data. That was the case for V0097 (the definitions' tenant_id_ moves from the UUID to the slug, across four Flowable tables) and V0098 (robots.parameters becomes jsonb, and anything that is not a JSON object is nulled).

./deploy/rehearse-migration.sh --dump backup.sql

The script starts a throwaway Postgres, restores the dump, prints the state before, applies the pending migrations and prints the state after. The container is removed at the end, with or without an error, and the dump is never written to. It takes pg_dump in plain or custom format, and --migration V0097 limits which ones run.

The report answers three questions: how many rows each table moves, how many match no tenant at all and therefore stay as they are, and how many robots would lose their parameters for not being a JSON object. That last number is the one worth a conversation before the deploy: those robots will need reconfiguring from the screen.

Initial Super Admin

The initial user is created by the V0001__init.sql migration, which runs by itself on first startup — there is no manual step to perform.

Field	Value
Email	admin@flowiagentic.local
Password	Admin@1234
Role	SUPER_ADMIN

The email has no hyphen (flowiagentic, not flow-iagentic) — the database name does, which is where the confusion comes from. Logging in with the hyphenated version returns invalid credentials.

Change it immediately in production

The default password is public knowledge and must be changed before the application is exposed.

Production build (Frontend)

cd frontend
npm run build

The static files land in frontend/dist/ and can be served by any web server (nginx, Apache, CDN).

CI/CD

CI runs on GitHub Actions, with the workflows in .github/workflows/. There is no azure-pipelines.yml in the repository.

Workflow	What it does
ci.yml	Validation on every push to master/main and on every pull request
deploy-gcp-backend.yml	Deploys the backend to GCP
security.yml	Semgrep and OSV on every push; Dependency-Check and SBOM twice a week
deploy-gcp-frontend.yml	Deploys the frontend to GCP
deploy-gcp-docs.yml	Deploys this documentation
deploy-gcp-landing.yml	Deploys the marketing site

ci.yml has two parallel jobs:

1. Backend (Java 21) — brings up a pgvector/pgvector:pg16 Postgres as a service and runs mvn verify, excluding S3StorageProviderTest and AttachmentServiceTest, which depend on external storage. mvn verify does not start the application and does not write openapi.json: that artifact lives in the openapi profile, described in REST API.
2. Frontend (TypeScript) — npm run build, npm run test:run, npm run i18n:check and npm run lint:check. The type step is the full production build, not tsc --noEmit: on its own it skips the project references and lets through errors the production image catches.

i18n:check is blocking: one new string missing a translation in pt-BR, en or es fails the build.

lint:check blocks differently. ESLint carries old debt (368 findings), and failing every build over it would only teach people to skip the step; so the count per rule is frozen in frontend/scripts/lint-baseline.json. Old debt passes; a rule that grows fails; and a rule nobody was violating has a ceiling of zero, so a new category of problem cannot arrive quietly. Fixed some findings? npm run lint:check -- --update-baseline locks the gain in — the command refuses to raise a ceiling, only to lower it.