Deployment and environment

Running locally (development)

Prerequisites

• Java 21+
• Maven 3.9+
• Node.js 20+ and npm
• PostgreSQL 16 with the pgvector extension, locally or via Docker. docker-compose.yml uses the pgvector/pgvector:pg16 image; a PostgreSQL 16 without the extension fails creating vector_store at startup

1. Start the database

docker compose up -d db

This starts only the PostgreSQL container, on port 5432.

2. Run the backend

cd backend

export DB_HOST=localhost
export DB_PORT=5432
export DB_NAME=flow-iagentic
export DB_USERNAME=flow-iagentic
export DB_PASSWORD=flow-iagentic
export JWT_SECRET=dev-secret-key-min-256-bits-for-hmac-sha256-algorithm

mvn install -DskipTests
mvn spring-boot:run -pl core -Dspring-boot.run.profiles=dev -Dcheckstyle.skip=true

The backend comes up at http://localhost:8080.

Two traps in this step — mvn package instead of mvn install, and the container that does not reload by itself — are covered under Local environment diagnostics, at the end of this section.

The third artifact: the runner agent

The same mvn install also produces backend/runner-agent/target/flowia-runner-agent.jar. It is neither the application nor RobotRunner.jar: it is the long-lived agent installed on a customer's machine, which fetches work from the platform and spawns the sandbox there.

The agent must not depend on core

runner-agent runs on a machine the platform does not administer. A dependency on core would carry the per-tenant datasource routing, the Flowable engine and the database driver into the customer's network — and it would compile without breaking anything. AgentCarriesNoCoreTest reads the resolved classpath and fails when that happens; it is what makes the violation surface in the build rather than on a customer's server.

The loop closes end to end: the platform assigns, the agent collects from /api/internal/runners/work, downloads both JARs verifying their SHA-256, spawns the child and reports the outcome back. The details are in specs/automation/robot-runners.md.

The same build also assembles backend/robot-runner/target/RobotRunnerPython.zip (the bootstrap and the flowia_robot SDK for Python robots), and the backend image copies it to /app/runner/ next to RobotRunner.jar. The platform serves it at GET /api/internal/runners/artifacts/runner?runtime=PYTHON, with its SHA-256 in a header, from app.robots.python-runner-path (default runner/RobotRunnerPython.zip).

The agent accepts Python robots when it finds Python 3.10+ with venv and pip:

Option	Effect
--python <path>	Interpreter to use. Without it, python3 on the PATH (on Windows, python3 then python). Without Python 3.10+ the agent still starts; it just does not declare Python and receives no Python robot
--python-env NAME=VALUE	Environment variable the operator lets through to Python robots (repeatable), for example PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright. A robot never sets variables; PATH, HOME, temp variables, BASH_ENV, ENV and the PYTHON*, LD_*, DYLD_* and PIP_* families are refused at startup. Values never reach the agent's log
--python-max-memory-mb <n>	RLIMIT_AS of each Python robot, default 2048. It only applies on Linux; on macOS and Windows it is a warning in the execution log, not a limit. A robot with a browser needs more, because Chromium reserves far more virtual memory than it uses

The runner has to be registered on the screen before the agent will start, by pasting the public key --init prints. The commands and the screen walkthrough are in Robots → Registering a runner.

3. Run the frontend

cd frontend
npm install
npm run dev

The frontend comes up at http://localhost:3000 (vite.config.ts → server.port). Serving it on another port means adding that port to app.cors.allowed-origins, or every call answers Invalid CORS request.

Local environment diagnostics

install is not optional, and package will not do

The reactor has three modules: core is the application, robot-runner is the sandbox the robots run in, and runner-agent is the agent installed on a customer's machine. core depends on robot-runner and resolves that dependency through the local ~/.m2 repository — not through the sibling's target/ directory. runner-agent depends on neither, on purpose.

mvn package builds both and publishes nothing to ~/.m2, so the cache keeps the previous robot-runner. mvn install is what refreshes it. The failure does not surface in the build: it surfaces at the next boot, as a compilation error inside the container, pointing at code you were certain you had fixed.

If you use the Docker environment, the host's ~/.m2 is mounted into the container (docker-compose.dev.yml), so the cache is shared — and so is the problem.

The same install leaves RobotRunner.jar in backend/core/runner/, where the local backend takes the sandbox from. The jar is not versioned: in a fresh clone, no robot runs before the first mvn install.

You ask for the container restart, the compiler does not

docker-compose.dev.yml mounts ./backend/core/target into the container, and that mount is the hot-reload mechanism: you compile on the host and the container sees the new .class files at once.

It does not reload on its own. When the build is finished, ask for the restart:

touch backend/core/target/classes/.reloadtrigger

DevTools then applies everything that changed since the last restart in one go (~1-2s, without restarting the JVM). The same applies to a build made inside the container (docker compose exec app mvn compile -q -pl core).

What holds the automatic restart back is spring.devtools.restart.trigger-file: .reloadtrigger, in application-dev.yml — the dev profile only, never prod or staging. SPRING_DEVTOOLS_RESTART_ENABLED=false would not do: spring-boot:run installs the RestartInitializer and DevTools restarts regardless, saying so in the log. The trigger file, in contrast, is read later, by the autoconfiguration — the watcher stays up and simply stops treating a new .class as a reason to restart.

Without the trigger-file, any build restarts the container — including a single-class mvn test, and including mid-build, while the classes are still half-written. The symptoms do not resemble the cause: NoClassDefFoundError, a @Configuration that never registers, a preflight refusing what it should accept.

If you touch the file and nothing happens, check that it is under target/classes (a mvn clean deletes it; touch recreates it) and that the container log shows Restarting due to ....

Environment variables (Backend)

Variable	Description	Default (dev)
DB_HOST	PostgreSQL host	localhost
DB_PORT	PostgreSQL port	5432
DB_NAME	Database name	flow-iagentic
DB_USERNAME	Database user	flow-iagentic
DB_PASSWORD	Database password	flow-iagentic
JWT_SECRET	Secret key used to sign JWTs	—
APP_ENCRYPTION_KEY	Encrypts third-party credentials at rest: AI provider keys, webhook signing secrets, secret tenant variables, the SSO client secret and mail settings. Losing it makes those unreadable, and it cannot be rotated once secrets are stored.	—
SECRETS_INJECT_INTO_PROCESS_VARIABLES	With false — recommended — secret variables never reach a process instance, so Flowable's runtime and history tables never hold a credential. Expressions use secrets.get(execution, 'key') instead.	true
AI_HTTP_TOOL_DEFAULT_POLICY	With deny — recommended — an agent with no host list makes no outbound call	allow
AI_STEP_CORE_SIZE	The installation's AI capacity: concurrent AI steps in steady state, on the dedicated pool where a step waits for the provider. This is the number to quote	8
AI_STEP_QUEUE_CAPACITY	AI steps that may wait for a slot before the pool refuses. Refused, a step is closed as a failure and is available for reprocessing	100
AI_STEP_PROVIDER_TIMEOUT_SECONDS	How long a step waits for the provider before giving up. A reservation's lease is twice this	120
AI_RATE_LIMIT_REQUESTS_PER_MINUTE	Requests per minute each node sends to one model (model + provider key), counting every tool round. 0 = no rate cap. Set your tier's RPM divided by the number of nodes	0
AI_RATE_LIMIT_MAX_CONCURRENT_CALLS	Concurrent AI step calls to one model, per node. 0 = no cap beyond the pool. A step that finds no free seat goes back to the queue at once, without holding its thread — a model at its cap does not delay steps of other models	0
AI_RATE_LIMIT_MAX_WAIT_SECONDS	How long a step holds its thread waiting for its turn in the per-minute rate before going back to the queue. Keep it well below the lease	30
AI_RATE_LIMIT_MAX_RETRY_DELAY_SECONDS	Ceiling on the wait before another attempt — the provider's hint on a rate-limit 429, or the backoff after a 5xx. It also caps the model's pause after a 429: a one-hour Retry-After pauses only up to this	120
AI_RATE_LIMIT_MAX_HOLD_MINUTES	How long, in total, a step may wait for capacity without calling the provider before it becomes a failure	30
AI_RATE_LIMIT_QUOTA_EXHAUSTED_PAUSE	How long a model stays marked as out of quota on this node after a provider quota 429, when the provider does not say when it comes back. Meanwhile class-bound agents go straight to the next model	60m
FLOWABLE_EXECUTOR_CORE_SIZE	Concurrent Flowable jobs in steady state — timers, async continuations and the AI step's completion. Since 2026-09-09 it is not the AI ceiling	2
FLOWABLE_EXECUTOR_MAX_SIZE	Job pool ceiling. The extra thread is only created once the 100-slot queue is full, and by then jobs are already being rejected — it is not capacity	3
JWT_EXPIRATION_MS	Access token expiry (ms)	3600000 (1h)
SPRING_PROFILES_ACTIVE	Active Spring profile	dev
APP_NODE_ROLE	all, api or worker. Decides whether the node acquires engine jobs, runs cluster-wide scheduled work and applies the shared Flyway set. See "Node roles"	all
APP_NODE_AFFINITY_ENABLED	With true, each tenant has one owning node and the node only acquires jobs of the tenants it owns. See "Tenant affinity"	false
APP_NODE_AFFINITY_HTTP	Declares that the router in front hashes on X-Tenant-ID. It changes no routing — only the connection arithmetic, which the application cannot observe on its own	false
APP_DATABASE_MAX_CONNECTIONS	The database server's max_connections. 0 means "not stated" and turns the comparison off	0
APP_DATABASE_EXPECTED_TENANTS	How many active tenants the installation expects. Feeds the connection ceiling	0
APP_DATABASE_EXPECTED_PODS	How many backend pods the installation expects	1
APP_DATABASE_BUDGET_ENFORCE	With true the node refuses to boot when the ceiling passes APP_DATABASE_MAX_CONNECTIONS, instead of only warning	false
APP_DATABASE_ROLE_ENFORCE	With true the node refuses to boot when its database role is rolsuper or rolbypassrls, instead of only warning. See "The database role"	false
AI_FAIL_ON_EMPTY_RESPONSE	Fail the task when the agent returns an empty response, instead of writing a blank value	true
AI_LOG_RAW_RESPONSE_ON_EMPTY	Log the raw provider response when it arrives with no candidate, to diagnose why it was blocked	true
AI_TASK_FAILURE_RETENTION_DAYS	Days an AI task failure record is kept before the daily purge	90
FORM_PASSWORD_HISTORY_RETENTION_DAYS	Days a password typed into a form stays in the history, already encrypted, before the nightly sweep wipes it	7
FLOWABLE_HISTORY_RETENTION_DAYS	Days the history (BPMN and CMMN) of an ended instance stays in the database before the nightly purge. A running instance is never touched. See "Data retention"	365
FLOWABLE_HISTORY_RETENTION_ENABLED	With true, the nightly prune deletes the Flowable history of instances ended more than FLOWABLE_HISTORY_RETENTION_DAYS ago. Off, nothing is deleted	false
AI_STEP_RESERVATION_RETENTION_ENABLED	With true, deletes AI step reservations that ended more than AI_STEP_RESERVATION_RETENTION_DAYS ago	false
ROBOT_EXECUTION_RETENTION_ENABLED	With true, deletes robot executions that ended more than ROBOT_EXECUTION_RETENTION_DAYS ago, with their log	false
FLOWABLE_HISTORY_LEVEL	History level of both engines: none, activity, audit or full. activity drops the variable history the field miner and the ended-instance screen read	audit
AI_STEP_RESERVATION_RETENTION_DAYS	Days a terminal AI step reservation (COMPLETED/FAILED) is kept. One whose instance still runs is always kept	30
ROBOT_EXECUTION_RETENTION_DAYS	Days a finished robot execution is kept, with its log. The latest execution of each robot is always kept	180
RETENTION_ALLOW_TENANT_CHOICES	With false, the retention choice made by each tenant's Admin is ignored and only the installation window applies: no tenant deletes anything while the default is off. The retention screen says so and locks its options. Helm: app.retention.allowTenantChoices	true
ROBOT_EXECUTION_RETENTION_DELETE_LOG_OBJECTS	With false, the execution leaves the database and its sealed log stays in the bucket, left to the bucket's lifecycle rule	true
SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE	The node's only pool, and therefore its whole budget: (pool + 1) × pods. It has a floor — see "The pool floor, and why it does not go lower on its own"	10
TENANT_POOL_MAX_SIZE	The ceiling one tenant may hold inside the shared pool. It is not a pool and opens no connection: it is the bulkhead that keeps one company from consuming everyone else's connections. The default comes from the node role: 7 on all/worker, 3 on api. Same floor	7
APP_STORAGE_TYPE	S3 or LOCAL. Outside the dev and test profiles the application refuses to boot on LOCAL	LOCAL (only usable in dev)
APP_STORAGE_S3_ENDPOINT	S3-compatible endpoint. Empty means AWS S3 itself	—
APP_STORAGE_S3_REGION	Bucket region	us-east-1
APP_STORAGE_S3_BUCKET	Bucket holding attachments, profile pictures and robot JARs. Required outside dev/test	flow-iagentic
APP_STORAGE_S3_ACCESS_KEY	Access key. Required outside dev/test	—
APP_STORAGE_S3_SECRET_KEY	Secret key. Required outside dev/test	—
RAG_INGESTION_MAX_FILE_SIZE	Maximum size of a knowledge base document. It applies only to the knowledge base upload route, which reads its own multipart with this limit (with or without Content-Length); other uploads stay at UPLOAD_MAX_FILE_SIZE. The proxy in front must accept the same size	50MB
UPLOAD_MAX_FILE_SIZE	The container's multipart limit, for every other upload (attachments, profile photo). Above it, 413. It does not apply to the robot artifact upload, which reads its own body	10MB
ROBOTS_MAX_ARTIFACT_SIZE_MB	Ceiling of the robot artifact upload (JAR or Python package), in MB, streamed to a temporary file. It applies only to POST /api/a/robots/{key}/artifact and /jar. The proxy must accept the same size: deploy/nginx.prod.conf already carries the route at 201 MB; on Helm, set nginx.ingress.kubernetes.io/proxy-body-size in ingress.annotations	200
ROBOTS_MAX_CONCURRENT_UPLOADS, ROBOTS_UPLOAD_TEMP_DIR	How many robot artifact uploads one instance serves at once (above that, 429) and the folder where the body is kept while it is checked (empty: the system temporary folder). Each upload can take up to ROBOTS_MAX_ARTIFACT_SIZE_MB on disk	4, empty
ROBOTS_PYTHON_ENABLED	Accepts Python packages on the robot upload. Leave it off until the runners have Python 3.10+	false
ROBOTS_PYTHON_RUNNER_PATH (app.robots.python-runner-path)	Where the backend reads the RobotRunnerPython.zip it serves to runners and that the robots screen offers under Download Python SDK. The image already copies it to /app/runner/; on a hand-built installation, point it at backend/robot-runner/target/RobotRunnerPython.zip	runner/RobotRunnerPython.zip
ROBOTS_PYTHON_MAX_ENTRIES, ROBOTS_PYTHON_MAX_UNCOMPRESSED_MB, ROBOTS_PYTHON_MAX_COMPRESSION_RATIO	Limits of the package check against a decompression bomb: entries, uncompressed MB, and compression ratio per file	20000, 1024, 100
RAG_INGESTION_MAX_PAGES	PDF pages, slides or TIFF frames per document; above it the version fails with TOO_LARGE before any embedding	2000
RAG_INGESTION_MAX_ROWS	Rows adding up every sheet of a document	200000
RAG_INGESTION_MAX_EXTRACTED_CHARS	Extracted characters per document. Also guards against a compressed file that explodes when opened	5000000
RAG_INGESTION_TIMEOUT	Maximum extraction time per document	15m
RAG_INGESTION_SHEET_ROWS_WARNING	A sheet with more rows than this carries the "code table — use the CMS" warning	5000
RAG_INGESTION_THREADS	Concurrent ingestions per node (rag-ingest- pool). A version that finds no slot stays PENDING and the 5-minute sweep queues it again	2
RAG_OCR_ENABLED	Empty: OCR is on only if the tesseract binary answers on the node. false turns it off even with the binary present; true with the binary missing stays off, with a log warning	empty
RAG_OCR_LANGUAGES	Tesseract languages, joined by +. The default image ships por, eng and spa (tessdata_fast)	por+eng+spa
RAG_OCR_MAX_PAGES_PER_DOCUMENT	Pages that go through OCR in one document; the rest is left out and the version is PARTIAL	200
RAG_OCR_PAGE_TIMEOUT_SECONDS	Per-page deadline; when it runs out, the Tesseract process is killed	60
RAG_OCR_CONCURRENCY	Concurrent OCR processes per node	1

Enterprise deployment (on-premise / self-hosted)

Flowi Agentic is designed to run inside the customer's own private cloud infrastructure (AWS VPC, Azure, local VM). Because of the BYOK (Bring Your Own Key) model the instance runs isolated and sends no telemetry or billing data to Flowi Agentic Inc. The only thing configured is the on-premise serial key that unlocks the licence.

# Build and start the cluster (standard node)
docker compose up --build -d

For the customer's Kubernetes cluster, use the Helm chart in the next section rather than Docker Compose.

A second replica works, and now it can have a role of its own

Scheduled work does not double up: robot CRONs, e-mails and cleanups take a database lease before running, so only one node runs each of them.

The three obstacles that used to fail silently are gone:

• Real time reaches both pods. What one node publishes travels through Postgres LISTEN/NOTIFY and is delivered to every pod's STOMP sessions. No new broker to install, and no session affinity required.
• A Java Delegate uploaded to one pod appears on the others within app.delegates.reconcile-interval-ms (60 s by default), with no restart.
• Rate limits belong to the installation. The login limiter and each API key's budget are counted in a shared table, so the number on the screen is the number that holds with one pod or with five.

Flyway also left the boot path: each tenant schema migrates on that tenant's first request and a leased sweep catches the rest, so boot time no longer grows with tenant count.

Runners scale on their own, and by a different knob. runner.replicaCount is a StatefulSet: each replica is a distinct runner with its own keypair on its own volume. Since a shared runner runs one execution at a time, it is that number — not backend.replicaCount — that decides how many robots run at once.

The second pod can now have a role. APP_NODE_ROLE=api serves HTTP without acquiring jobs, without running cluster scheduled work and without applying the shared Flyway set, on a smaller per-tenant ceiling; APP_NODE_ROLE=worker does the opposite and answers only health over HTTP. Without the variable a node is all, which is the behaviour it always had. See "Node roles".

What decides the pod count is the connection budget, and it multiplies per pod only: (sharedPool + 1) x pods — the parenthesis is what one pod costs, and the +1 is the connection it holds open to listen for events. Since 2026-09-09 there is no per-tenant term in that sum. The application does the arithmetic and prints it on every boot; state APP_DATABASE_MAX_CONNECTIONS and APP_DATABASE_EXPECTED_PODS and it warns when the number does not fit (APP_DATABASE_EXPECTED_TENANTS is still printed on that line, precisely to show that it does not move it). See "Connection budget".

The ReadWriteOnce volume is still the obstacle the chart detects and refuses. With S3 required it is only a cache: persistence.enabled=false takes that one out of the way.

Docker services

Service	Container	Port
Database	flow-iagentic-db	5432
Backend	flow-iagentic-app	8080

Frontend

The frontend does not run in Docker by default — run it locally with npm run dev.

docker-compose brings up no runner; the chart does

docker-compose neither brings up nor registers a runner: here a robot only executes after someone registers a runner on the screen and starts the agent on its machine — see Registering a runner is part of bringing the installation up.

Helm has been different since 2026-08-29. The chart brings up runner.replicaCount runners (1 by default), each generating its own keypair inside its own volume and enrolling itself as GLOBAL. The private key is born on the pod that signs with it and never travels. A runner scoped to a named tenant still requires someone pasting a public key on the screen.

Kubernetes deployment (Helm)

The chart lives in helm/ and installs backend + SPA + runners into the customer's cluster. It pulls no dependencies: there is no remote subchart, so the install works in a cluster with no route to the internet, as long as the images are reachable.

Requirements

Kubernetes	1.23+
Helm	3.8+
Ingress controller	any — or none, with ingress.enabled=false
PostgreSQL	16 with pgvector, embedded or external
Object storage	required — any S3-compatible endpoint, plus a bucket and a key that can read, write and delete in it

Evaluation — embedded database, your bucket

cd helm
helm install flowia ./flowia -n flowia --create-namespace --wait --timeout 20m \
  --set storage.s3.endpoint=https://s3.customer.com \
  --set storage.s3.bucket=flowia-attachments \
  --set storage.s3.accessKey=... --set storage.s3.secretKey=...
kubectl -n flowia port-forward svc/flowia-frontend 8080:80

Comes up with embedded PostgreSQL and generated secrets. The four storage flags are not optional: object storage is a requirement, and an install without them fails at render time rather than when somebody attaches a file. The first boot runs Flyway on the shared schema and on every tenant schema — it takes minutes, and the startupProbe waits up to 600s before declaring the pod dead. Do not shorten that thinking it has hung.

port-forward does not carry real time

The task inbox's live updates use a WebSocket at /ws, and what routes /ws is the Ingress — not the frontend's nginx. Without an Ingress the screen works but does not refresh itself.

Production

helm install flowia ./flowia -n flowia --create-namespace \
  -f examples/production.yaml --wait --timeout 20m

helm/examples/production.yaml is the shape, not the values: the customer's database, the customer's ingress, the customer's managed secrets.

The database is an on/off switch

# embedded (default) — pgvector StatefulSet + PVC, for evaluation
--set postgresql.enabled=true

# external — the database the customer already runs
--set postgresql.enabled=false \
--set externalDatabase.host=pg.customer.com \
--set externalDatabase.database=flowia \
--set externalDatabase.username=flowia \
--set externalDatabase.existingSecret=flowia-db

The embedded database has no backup, no failover and no pooler — it exists so an evaluation installs in one command. The external one must allow CREATE EXTENSION vector (Spring AI creates it at boot) and must let the application create schemas, because every tenant is a schema.

With externalDatabase.host missing, helm template fails immediately with the written message — rather than a CrashLoopBackOff half an hour later.

Object storage is a requirement, not an option

Every installation requires S3. storage.type defaults to S3, and storage.s3.endpoint and storage.s3.bucket have no defaults: leave either one empty and helm template fails immediately with a written message — the same way it fails without externalDatabase.host. A helm install with no values at all is not valid.

--set storage.s3.endpoint=https://s3.customer.com \
--set storage.s3.bucket=flowia-attachments \
--set storage.s3.accessKey=... --set storage.s3.secretKey=...
# or: --set secrets.existingSecret=flowia-secrets, carrying S3_ACCESS_KEY and S3_SECRET_KEY

storage.type=LOCAL is development and test only. There the volume at /datais attachment storage: one ReadWriteOnce disk with no backup, no replication, no lifecycle policy and no second reader — losing it loses every attachment in the installation. The application refuses to boot on LOCAL outside the dev and test Spring profiles, so a chart that renders LOCAL will not produce a running pod.

storage.type	The volume at /data	persistence.enabled=false
S3 (default)	is only a cache of robot JARs	allowed — becomes an emptyDir
LOCAL (dev/test)	is attachment storage	refused by the chart

A robot's JAR is written to durable storage on upload and pulled down on demand when it is missing from disk, which is why the cache can be ephemeral. persistence.enabled defaults to true so a fresh pod does not have to fetch every JAR from the bucket before its first execution; on S3, turning it off costs that latency and nothing else.

The chart does not deploy MinIO. Point storage.s3.endpoint at the object storage the customer already has, including a MinIO they run themselves, which is their licence and their decision. An empty endpoint is refused rather than treated as "AWS S3": an on-premise release that quietly uploaded the company's attachments to Amazon is worse than one that does not start.

File retention is a bucket rule, not a job of ours

Expiring an old attachment is not the platform's business: it ships no purge job for stored files and will not grow one. Expiring an old object is a bucket lifecycle rule (or your appliance's ILM), configured by whoever owns the bucket, next to the retention and legal-hold policy the business already has.

Deleting an attachment in the product deletes its object immediately — that is deletion on demand, not retention. The same holds for the sealed log of a robot execution the data retention below deletes: the object goes with the row that points at it, and nothing else does. If you do write an age-based rule, scope it so it cannot expire objects the database still references, or the attachment comes back to the user as a download error.

Data retention: what the platform deletes on its own

Three things grow with use and, until now, nothing deleted them: Flowable history (the act_hi_* and act_cmmn_hi_* tables, in public, shared by every tenant), AI step reservations and robot executions. Each now has one default window for the whole installation and a nightly job that purges what is past it. All three jobs ship off: nothing is deleted until the operator turns each one on (*_RETENTION_ENABLED=true or enabled: true in Helm), because deleting cannot be undone and nothing is archived first:

What	Default window	Goes	Never goes
Flowable history (BPMN and CMMN)	365 days after the instance ended	the ended instance, with its activities, tasks, variables and links	a running or suspended instance, or one waiting on a dead-letter job — it has no end time
AI step reservations	30 days after becoming COMPLETED or FAILED	the terminal reservation	a RESERVED/COMPLETING reservation, and any reservation of an instance that still runs
Robot executions	180 days after the execution finished	a SUCCESS, FAILED, TIMEOUT or CANCELLED execution, with its log lines and its sealed log in the bucket	an execution in progress or pending, the latest execution of each robot, and an execution whose log the bucket refused to delete (retried the next night)

The three jobs run between 04:45 and 05:15, after the other cleanups and the 04:00 mining run. They take the job_leases lease — one node purges per night, and an api node never does —, walk every tenant with the tenant filter on every statement, and delete in batches. Every run logs how many rows left per tenant and adds them to the flowi.retention.removed counter (tag table).

In Helm the values live under app.retention:

app:
  retention:
    flowableHistory:    { enabled: false, retentionDays: 365, minRetentionDays: 90, maxRetentionDays: , batchSize: 500,  maxBatchesPerRun: 200, historyLevel: audit }
    aiStepReservations: { enabled: false, retentionDays: 30,  minRetentionDays: 7,  maxRetentionDays: , batchSize: 1000, maxBatchesPerRun: 100 }
    robotExecutions:    { enabled: false, retentionDays: 180, minRetentionDays: 30, maxRetentionDays: , batchSize: 500,  maxBatchesPerRun: 100, deleteLogObjects: true }

Per-tenant window, within the installation bounds

Each tenant's Admin chooses its own window under Administration → Data retention (see the screen): installation default, off, or N days. What the operator sets here is the default — what applies to a tenant that chose nothing — and the bounds:

Value (Helm / variable)	What it does	Default
minRetentionDays / FLOWABLE_HISTORY_RETENTION_MIN_DAYS, AI_STEP_RESERVATION_RETENTION_MIN_DAYS, ROBOT_EXECUTION_RETENTION_MIN_DAYS	Shortest window a tenant may choose. Protects what the installation must keep	90 / 7 / 30
maxRetentionDays / FLOWABLE_HISTORY_RETENTION_MAX_DAYS, AI_STEP_RESERVATION_RETENTION_MAX_DAYS, ROBOT_EXECUTION_RETENTION_MAX_DAYS	Longest window a tenant may choose. Empty, no maximum. With a maximum, "off" is no longer an option for the tenant	empty

• enabled: false turns off the default, not the screen: a tenant whose Admin chose N days is purged with that window even with the default off. Nothing changes on upgrade — no tenant has a choice until its Admin saves one.
• A minimum below 1, a maximum below the minimum, or — with the default on — a retentionDays outside the bounds refuses the boot.
• If you tighten the bounds later, a saved choice outside them is adjusted to the bound at run time, with a WARN log line naming the tenant and the table; the stored value is not rewritten.
• Every change of a tenant's window is kept in the tenant_retention_changes table (who, which table, from what to what, when).

Outside Helm, the variables are the ones in the table above, plus each one's *_BATCH_SIZE, *_MAX_BATCHES and *_CRON (FLOWABLE_HISTORY_RETENTION_BATCH_SIZE, AI_STEP_RESERVATION_RETENTION_MAX_BATCHES, ROBOT_EXECUTION_RETENTION_CRON…).

What to decide before upgrading:

• Deleting is not archiving. There is no export before the purge. If a customer needs the history of an ended process for longer than the window — audit, regulation —, raise the window or turn the table off (enabled: false) before the first night.
• The first night meets the backlog. batchSize bounds one DELETE; maxBatchesPerRun bounds one night per tenant and table (with the defaults, up to 100,000 history instances per tenant per night). A larger backlog drains over several nights instead of holding the lease in one long run.
• The field miner reads history. A mining window (timeWindowMonths) longer than the history retention only sees what is still kept.
• A window or batch below 1 refuses the boot: to delete nothing, use enabled: false — and, if no tenant may keep less than the default, a minRetentionDays equal to it.
• Flowable's native cleaning timer (enableHistoryCleaning) stays off on purpose: the job it creates carries no tenant, and this platform refuses tenant-less jobs. The purge uses the engine's own cleaning predicate (instances ended before the cutoff), narrowed by tenant and with each tenant's window.

To measure the effect, compare the dashboard aggregate's timing before and after — it counts through act_hi_taskinst and is the cheapest gauge of how large the history is.

Migrating an installation that already runs on LOCAL

The upgrade does not move the files. They are on the PVC; neither Helm nor the backend copies anything. Nothing in this change deletes an attachment, and nothing in it copies one either. In order:

1. Provision a bucket on the object storage the customer already runs, plus a key that can read, write and delete inside it.
2. Copy the files out of the volume. They live under persistence.mountPath — /data/attachments by default — laid out as <directory>/<fileName>, which is exactly the S3 key S3StorageProvider would have written. An mc mirror or aws s3 sync of /data/attachments/ into the bucket root reproduces the layout. Copy from a pod with the volume mounted, or from a snapshot; the backend may keep running.
3. Rewrite attachments.storage_path. This is the step that is easy to forget and impossible to skip: under LOCAL the column holds an absolute path (/data/attachments/<directory>/<fileName>), under S3 it holds the key (<directory>/<fileName>). attachments is a per-tenant-schema table, so the update runs once in every tenant_<slug>, not once globally.
4. Only then upgrade, with the storage values set.

There is a shortcut for a small installation, and it is a shortcut: re-upload the attachments through the UI and let the platform write them to S3 itself. It works, it produces new rows with new keys, and it does not scale past a few dozen files.

Memory: robots do not run inside the backend container

No robot opens in the application's JVM: every execution is assigned to a registered runner, which is another machine (or another pod) running the agent. backend.resources sizes the platform alone, and the sandbox heap ceiling belongs to the agent, on the agent's machine — the chart has no key for it.

Whoever sizes memory for robots sizes the runner's machine. An installation with no runner registered spends no memory on robots — and runs none either.

Registering a runner is part of bringing the installation up

Without at least one runner registered and answering, executions stay PENDING forever, with the reason written on the row itself. The platform runs no robot of its own.

With Helm the chart already handles this: it brings up runner.replicaCount runners (1 by default) that generate their own key inside their own volume and enrol themselves as GLOBAL. Only enrolment from inside the release does that, and it can only ever create a GLOBAL runner — holding the token buys a share of the shared pool, not a tenant's robot work.

The runner image (backend/runner-agent/Dockerfile) is now glibc and ships Python. It uses eclipse-temurin:21-jre-noble with python3 and python3-venv (Python 3.12; pip comes from the venv's ensurepip), no longer the Alpine base: almost every binary PyPI wheel (numpy, pandas, lxml) is manylinux, and it does not install on a musl interpreter. The image grows by about 60 to 80 MB, the user is still 1000, and neither the chart nor docker-compose.dev.yml needs new configuration: the agent finds python3 on the PATH. Browsers (Playwright, Selenium) are not in the image (that would be another ~400 MB): whoever needs them starts from this image, installs the browser and passes the path with --python-env. The chart does not yet have a field for passing extra arguments to the agent.

Outside Helm, and for any runner scoped to a named tenant, registration is manual, and deliberately stays so: deciding which machine receives a tenant's code is a decision for a person (specs/automation/robot-runners.md → How the chart installs runners).

The step runs once per machine, and the commands and screen walkthrough are in Robots → Registering a runner. What belongs to whoever brings the installation up is app.api.base-url: the agent probes the platform's RPC address at startup and stops, naming the address, when it cannot reach it. localhost there is the platform itself, never the runner's machine. The chart fills this in on its own — the Ingress URL when there is one, the in-cluster Service otherwise — and app.apiBaseUrl in the values overrides it.

The runner must not be the application's own machine

Installing the agent on the backend's host puts the tenant's code back on the disk holding every other tenant's configuration, credentials and attachments — exactly the exposure that executing outside the application avoids. The chart does not prevent it; the choice belongs to whoever installs.

The operating envelope: what one installation holds

Before the arithmetic, the honest summary: this platform's scalability is bounded by the database, and that is a consequence of schema-per-tenant isolation, not a defect to fix. Every tenant gets its own schema, and that is what keeps one company from seeing another's data. What the database charges, however, no longer grows with the number of companies: since 2026-09-09 there is a single pool per node, and the cost belongs to the pod.

Three limits decide the size of an installation. Only one of them is HTTP, and it is not the tight one.

Limit	What fixes it	How it shows up
Database connections	(sharedPool + 1) × pods	FATAL: sorry, too many clients already, on a user's request
AI step concurrency	AI_STEP_CORE_SIZE (8 by default), per installation	the AI step queue grows; waiting increases, and a step refused by a full queue goes to reprocessing
HTTP	pod CPU and memory	rarely the bottleneck here

Sizing by the database. With the default (sharedPool 10) every pod costs 11 connections — the 10 of the pool plus the cluster-bus listener — and that number does not move with the number of customers. How many pods fit, leaving out the headroom Postgres reserves for superusers and your own psql session:

max_connections	Typical Cloud SQL tier	pods that fit, at any number of tenants
25	db-f1-micro	2
50	db-g1-small — the managed deployment's	4
100	1 vCPU / 3.75 GB	9
200	2 vCPU / 7.5 GB	18
400	4 vCPU / 15 GB	36

The arithmetic, so any row can be checked: (10 + 1) × pods.

Until 2026-09-09 this table had a column per installation shape and a tenant count in every cell, because each tenant opened its own HikariCP pool and the sum was (sharedPool + poolPerTenant × activeTenants + 1) × pods. The per-tenant pool was removed: one pool serves everyone, the search_path is applied when a connection is borrowed and the session is reset when it is returned. The next customer stopped moving the number.

The ceiling arrives with the pod — and that is a lever

Redundancy used to cost tenants: two pods on a db-f1-micro did not sustain even one active customer, and the third customer took the redundancy off a db-g1-small. Now the only factor is the pod, and its price is fixed: another pod costs 11 connections and brings another Flowable executor with it. That is why scaling out became the way to buy job throughput — with a pool per tenant it was not, because each pod also multiplied the term for the tenants it served.

The shared pool has a floor, and the floor is not negotiable

An AI step holds two connections at its peak (the quota deduction is REQUIRES_NEW). Since every connection comes from the same pool, it needs app.flowable.executorMaxSize × 2 + 1 — 7 with the shipped executor. The chart refuses the render below that and the application warns at startup, because the failure it prevents is not too many clients: it is a tenant that quietly stops finishing AI steps, with no log and no error. The same floor applies to app.tenantPool.maxSize, which is the ceiling one tenant may hold inside the shared pool.

What the shared pool costs, measured rather than assumed:

• ~1.8x per short query, on a bench with four tenants alternating on the same physical connection (SharedPoolCheckoutCostMeasurementTest, perf profile). That is ~51 µs of extra round trip per checkout (applying the search_path and resetting the session) and ~15 µs of re-planning, because PostgreSQL re-analyses a statement when the search_path changes. At AI-step scale the difference disappears: with 1 s and 3 s model calls the sustained throughput is identical.
• The bulkhead is a ceiling, not a reservation. A pool per tenant guaranteed by construction that one tenant could not consume another's connections. What replaces it is a per-tenant ceiling inside the shared pool (app.tenantPool.maxSize), exact per node — but a ceiling, not a reservation: one tenant can wait because of another's burst.

Tenant affinity: one owning node per tenant

app.tenantAffinity.enabled (off by default) gives each tenant one owning node, the only one that acquires its jobs. Affinity's connection argument died with the per-tenant pool: there is no per-tenant term left for it to stop multiplying, and the budget is identical with it on and off. The startup line says so rather than staying silent about a benefit that no longer exists.

What it still buys, and what it asks in return:

• Job ownership and plan locality. A node acquires only its own tenants' jobs, and the database sees the same tenant through the same node.
• The router in front must hash on X-Tenant-ID. The chart writes nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x_tenant_id" when app.tenantAffinity.http is on, and refuses the render when ingressClassName is not nginx — another controller would ignore the annotation silently.
• A node that dies gives its tenants back in 30–40 s (one heartbeat plus the TTL). During that window those tenants' jobs wait; nothing is lost and nothing fails, but they are late. The details, including what each node logs, are in specs/core/tenant-node-affinity.md.
• A single-pod installation does not change at all — with one destination there is nothing for a hash to spread.

Sizing by AI. One installation runs 8 AI steps at a time (AI_STEP_CORE_SIZE), and throughput is AI pool core-pool-size × 60 ÷ latency_in_seconds. With ~20 s calls that is 24 steps per minute — for the whole installation, not per tenant. Past that the 100-slot queue grows and each step waits longer; past the queue, a step is closed as a failure and left for reprocessing rather than parked in silence. That is arithmetic: the bench with injected latency (specs/core/flowable-job-pool.md) measured the old, 2-slot design.

The provider's limit is another ceiling, and usually the first one. The provider limits requests per minute and per day per project and model, not per key or tenant — Gemini 3.1 Pro on tier 1 allows 25 per minute and 250 per day. Each AI step makes one request per tool round (about five in a typical step). The model limiter holds a burst back before it turns into 429s:

• Set AI_RATE_LIMIT_REQUESTS_PER_MINUTE to the tier's RPM divided by the number of nodes that run AI. The limit is per node, in memory; two nodes at 25 each send up to 50 a minute. For one model, use app.ai.rate-limit.models[<model name>].requests-per-minute (and .max-concurrent-calls) in application.yml or SPRING_APPLICATION_JSON — a name with a dot needs the brackets.
• With nothing configured, the first rate-limit 429 pauses that model on that node for as long as the provider asked (Please retry in 35s, retryDelay, Retry-After), and the step comes back after the wait, within its own attempt budget.
• A step held by the limiter did not call the provider: it spends no attempt and writes no usage row. It only becomes a failure after AI_RATE_LIMIT_MAX_HOLD_MINUTES of waiting.
• An exhausted-quota 429 (daily quota, plan, insufficient_quota, no credit left) stays final: waiting seconds does not help. The reason under Failed says which of the two it was. It only counts as quota when the body says so; an unexplained 429 (from a proxy, for instance) is treated as a rate limit: short pause and another attempt.
• The queue shows in flowi.ai.model.limiter.waiting and flowi.ai.model.limiter.oldest.wait.seconds, per model, on each node.

The limiter makes the ceiling safe; it does not raise it. To raise it: a higher tier on the provider project, Vertex AI instead of AI Studio in production, and a Flash model for the steps that do not need Pro.

What this design does not solve, and will not

Said once, so nobody discovers it in production:

• A busy tenant can make another wait. Connections no longer grow with the number of customers, but there is one pool: the per-tenant ceiling keeps one company from taking everything, and it reserves nothing for a company that is idle. Heavy concurrent writing inside a single tenant is the case this design has not measured — recorded as a known limit in specs/core/connection-pool-budget.md.
• Flowable's tables are shared. The 53 ACT_* tables live in public, filtered by tenant in the query. A tenant whose history dwarfs the others affects everyone's query times: the isolation there is of data, not of performance.
• AI throughput belongs to the provider. No configuration of ours makes a 20-second call faster. What you choose is how many run in parallel — and since 2026-09-09 each holds a thread of the AI pool, but not a database connection while it waits.
• The numbers here are worst case. They add up every pool being full at once. A real installation usually sits below them — and "usually" is not something to size against.

The recipe, in three lines

1. Count the tenants that will be active at the same time, not the ones registered.
2. Pick the tier from the table above, with headroom for your psql session and for peaks.
3. Only then decide replicas: with two pods, redo the arithmetic — and if it does not fit, split the roles before raising the tier, because an api node costs 3 connections per tenant instead of 7.

Connection budget: only the pod multiplies

There is one HikariCP pool per pod (spring.datasource.hikari.maximum-pool-size). Every tenant request comes out of it: SchemaBoundDataSource points the borrowed connection at tenant_<slug> and resets the session when it goes back. An installation's connection demand is:

(sharedPool + 1) × pods

The + 1 × pods is the cluster-bus listener (LISTEN flowia_cluster), which is how a real-time update published on one pod reaches a browser connected to another. Fixed cost per pod, not per tenant.

There is no tenant term. APP_DATABASE_EXPECTED_TENANTS still exists and still appears on the startup line, but it enters no sum: it is there so the operator sees, side by side, the customer count they expected and a budget that does not move with it.

The chart templates this, and knows how to refuse

externalDatabase:
  poolMaxSize: 10       # the ONE pool of each pod (floor: app.flowable.executorMaxSize × 2 + 1)
app:
  tenantPool:
    maxSize: 7        # ceiling ONE tenant may hold inside that pool (TENANT_POOL_MAX_SIZE)
    apiMaxSize: 3     # the same ceiling on a node with the `api` role
  database:
    maxConnections: 200   # the server's max_connections. 0 means "not told", and nothing is checked
    expectedTenants: 8    # printed at boot; it does not enter the sum
    expectedPods: 0       # 0 derives it from backend.replicaCount + backend.worker.replicaCount
    budgetEnforce: true   # refuse the `helm template` instead of refusing the connection

With budgetEnforce: true a release that does not fit fails to render, printing the sum and the ceiling, instead of installing and meeting the limit as FATAL: sorry, too many clients already under load. expectedPods: 0 is the default and the right value: the chart counts the pods it creates itself, so the number the platform is told is the number the cluster runs. Fill it in only when pods outside this release share the same database.

The application prints the same arithmetic, and can refuse too

The chart's guard catches a release before it installs. The node catches a pod configured past the chart — through backend.extraEnv, a hand-edited manifest, or a database whose tier changed under it. On every boot the backend prints one line with the node's role and the decomposition, not just the total (the log itself is written in Portuguese):

Orçamento de conexões (papel=all, pior caso): (10 compartilhado + 1 bus) = 11 por pod x 2 pod(s) = 22 conexões, contra APP_DATABASE_MAX_CONNECTIONS=50. Não há parcela por tenant: ...

The numbers come from APP_DATABASE_MAX_CONNECTIONS and APP_DATABASE_EXPECTED_PODS, which the chart fills in from app.database.*. Past the ceiling it warns loudly with the overshoot and the levers that remain; with APP_DATABASE_BUDGET_ENFORCE=true the node refuses to boot instead of warning. The default is false on purpose: refusing by default would turn one wrong number in values.yaml into total unavailability, which is worse than the problem being reported.

The number is worst case, not current consumption

It is the ceiling the shared pool may reach; spring.datasource.hikari.minimum-idle decides how many of those connections stand up while the node is idle. Reading the warning as "the database is already full" leads to resizing the wrong thing.

The two ways out, and when each is the right one

When the sum does not fit two moves remain — and the one that disappeared is the news:

Move	When it is the right one	What it costs
Raise the database tier	When the installation needs more pods, for job throughput or for redundancy	Money, and a Cloud SQL maintenance window
Fewer pods (backend.replicaCount, backend.worker.replicaCount)	When the current throughput is enough and connections are what is short	Redundancy, or job throughput — every pod carries an executor
Split the roles	No longer a connection lever. An api node and a worker cost the same: one pool each. Splitting still pays for load isolation, not for connections	—

And the move that stopped being necessary: raising the tier before an onboarding. The next customer does not move the number any more; what moves it is the decision to run another pod, and that decision is yours rather than theirs.

Shrinking the pool has a floor

externalDatabase.poolMaxSize (and app.tenantPool.maxSize) do not go below app.flowable.executorMaxSize × 2 + 1. Below that the AI steps hold and wait on each other, and the symptom is not an error: it is the tenant no longer finishing steps. Lowering the pool requires lowering FLOWABLE_EXECUTOR_MAX_SIZE with it — and then the installation runs fewer AI steps in parallel.

Two corrections worth knowing about:

• Until 2026-09-03 this formula counted sharedPool once, and the shared pool is per pod — each pod is a JVM with its own. The old version under-reported a two-pod install by a whole pool and read "fits" where the truth was "does not".
• Until 2026-09-09 it carried a poolPerTenant × activeTenants term. That left with the per-tenant pool. If you find a copy of either old formula anywhere, it is wrong. Recorded in specs/core/connection-pool-budget.md.

What the managed deployment runs today

The managed deployment runs two pods against a Cloud SQL db-g1-small (max_connections ≈ 50), with SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE=10:

pods	arithmetic	against 50
2	(10 + 1) × 2 = 22	fits
3	(10 + 1) × 3 = 33	fits
4	(10 + 1) × 4 = 44	fits, with little headroom for your own psql session
5	(10 + 1) × 5 = 55	does not fit

On any row, at any number of tenants. Until 2026-09-09 the same instance carried two active customers and ran out on the third; and on the db-f1-micro it was until 2026-09-05, the first customer already did not fit.

Changing the tier restarts the instance

gcloud sql instances patch --tier drops the open connections. Doing it with a customer already on the platform means scheduling downtime with them. That is why a tier change is a pre-onboarding item and not an incident-response one.

Required TLS and backups on the managed instance

Two other properties of the managed instance changed in the same window on 2026-09-05:

Property	Value	What it forces
settings.ipConfiguration.sslMode	ENCRYPTED_ONLY	A cleartext connection is refused by the server, not silently downgraded. Anything connecting over IP needs sslmode=require in the JDBC URL; the Cloud SQL connector negotiates TLS on its own
Automated backups	daily, with point-in-time recovery	There is a restore point that does not depend on someone remembering to run gcloud sql backups create

Point-in-time recovery restores the instance, not one customer

It returns the whole server to an instant: every tenant_<slug> schema and public, with the ACT_* tables and the discriminator tables of every tenant together. There is no "roll tenant acme back to yesterday" down this path — using it to fix one customer reverts the others with it.

What does exist for a single tenant is the pair POST /api/admin/tenants/{id}/content-export / POST /api/admin/tenants/{id}/restore, which runs pg_dump -n tenant_<slug> and restores over that schema. That is not a tenant backup: the dump covers only that schema, so running instances, tasks, jobs, history, AI agents, robots, API keys and webhooks — all of which live in public — are left out. See Multi-tenancy for which tables live where.

The pool floor, and why it does not go lower on its own

externalDatabase.poolMaxSize is the obvious lever for fitting more pods, and it has a floor that is correctness, not tuning. The AI quota deduction runs in its own transaction (REQUIRES_NEW), which suspends the job's transaction without releasing its connection and takes a second one. Every AI step therefore holds two at its peak:

floor = app.flowable.executorMaxSize × 2 + 1
      = 3 × 2 + 1
      = 7

The + 1 is the connection that has to be left for HTTP traffic. The floor applies to the smaller of externalDatabase.poolMaxSize (the pool a job draws from) and app.tenantPool.maxSize (that tenant's ceiling inside it): 20 free connections do not help a tenant whose ceiling is 3, because its steps hold and wait on each other inside the ceiling.

Below the floor the symptom is not an error — it is the tenant no longer finishing AI steps. Each job holds its first connection and waits for a second one that only frees when another job finishes, and none does. What shows up in the log is connection timeouts on flowable-task-exec-* threads, with nothing saying "the pool is too small".

The lock-up itself starts earlier than the floor suggests, and the distance is deliberate. On the rig in specs/core/job-worker-nodes.md the pool stops making progress at executorMaxSize and resumes at executorMaxSize + 1 — 4 with today's executor. Between 4 and 7 nothing locks up: steps merely wait for a connection. Those three of margin cover what the rig does not model, the row lock on the tenant's single quota row, which makes concurrent deductions queue while each caller still holds its connection. The guard is written as × 2 + 1 because that is the worst case; + 1 is only where it stops being fatal.

That is why shrinking the pool alone is refused at chart render time, with the arithmetic in the message. Shrinking both together stays allowed, and is the configuration that buys another pod:

externalDatabase:
  poolMaxSize: 5       # brings a pod down to 6 connections
app:
  flowable:
    executorMaxSize: 2   # lowers the floor to 5
  tenantPool:
    maxSize: 5

What that trade gives up is the third thread, which ThreadPoolExecutor only creates once the 100-slot queue is already full — the last relief before a job is rejected rather than merely delayed. Sustained throughput does not change: it is executorCoreSize, still 2.

app.tenantPool.apiMaxSize sits outside this arithmetic. An api node acquires no jobs, so no thread on it opens the quota transaction and none ever holds two connections — that, and not extra tolerance, is what allows the pool of 3 there.

The full derivation and the measured configurations are in specs/core/job-worker-nodes.md.

The database role: ordinary, never superuser

The application connects with an ordinary login role. It owns the public schema and has CREATE on the database — enough for the migrations, for provisioning's CREATE SCHEMA tenant_<slug> and for backup's pg_dump/psql — and it does not have SUPERUSER, BYPASSRLS, CREATEROLE, CREATEDB or REPLICATION.

This matters for one direct reason: a superuser role ignores schema permission and ignores Row Level Security, including FORCE ROW LEVEL SECURITY. With it, no database-level defence — present or future — works, and the whole isolation between customers rests on application code alone. SUPERUSER also enables COPY … FROM PROGRAM, which runs a command on the database host.

Check your installation with this command, substituting the name if you use a different externalDatabase.username:

SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'flow-iagentic';

Or straight from a shell:

psql "$DATABASE_URL" -tAc \
  "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'flow-iagentic'"

The right answer is flow-iagentic|f|f. Any t means the installation has no database-level barrier between customers.

To fix it, as a database superuser:

ALTER ROLE "flow-iagentic" NOSUPERUSER NOBYPASSRLS NOCREATEROLE NOCREATEDB NOREPLICATION;
GRANT CONNECT, CREATE ON DATABASE "flow-iagentic" TO "flow-iagentic";
ALTER SCHEMA public OWNER TO "flow-iagentic";

The backend checks this itself on every boot and prints the line:

Role de banco: flow-iagentic (rolsuper=false, rolbypassrls=false). Nenhuma defesa no nível do banco é ignorada por esta conexão.

When the role carries too much privilege, the same line becomes a warning and names the attribute. APP_DATABASE_ROLE_ENFORCE decides what happens: false warns and boots — the default, because on an on-premise installation the role is the customer DBA's decision, and a platform that will not start gets rolled back, taking the warning with it; true refuses to boot.

Where each default sits, and why:

Where	Default	Reason
docker-compose.yml / .dev.yml	true	An environment we control, and the fix is docker compose down -v
docker-compose.prod.yml	false	An existing volume is not re-provisioned; migrate the role, then turn it on
Helm chart (app.database.roleEnforce)	false	The chart is the on-premise delivery. helm/examples/production.yaml turns it on, after the query above answered f and f

The chart's bundled database runs the application as a superuser

postgresql.enabled=true brings up a StatefulSet whose POSTGRES_USER is the application's own role — a superuser, by construction of the image. It exists so an evaluation can start in one command, and the chart README already says it has no backup, no failover and no pooler. This is one more reason not to use it for anything real: use externalDatabase with a role provisioned as above.

An existing database volume is not re-provisioned

initdb scripts only run against an empty volume. A development environment created before this change still connects as a superuser — the backend refuses to boot and says why. docker compose down -v recreates the database from scratch. That is development data; on a database with real data, use the ALTER ROLE above instead of deleting the volume.

Node roles: which pods answer HTTP and which execute jobs

One image, two jobs, chosen by configuration. The default is one node doing both, which is what every existing installation has — the split is opt-in and changes nothing for anyone who does not ask.

Role	Serves HTTP	Executes Flowable jobs	Cluster scheduled work and robot CRONs	Migrates the shared schema	Default TENANT_POOL_MAX_SIZE
all	yes	yes	on the elected node	yes	7
api	yes	no	no	no	3
worker	/actuator only	yes	on the elected node	yes	7

The variable is APP_NODE_ROLE, and the chart writes it into both Deployments. A node that boots without it is all — exactly the behaviour it always had, which is why no existing installation changes on upgrade. A misspelled role refuses to boot rather than becoming all and serving under a role other than the configured one.

backend:
  role: api          # this Deployment only answers HTTP
  replicaCount: 2
  worker:
    enabled: true    # and this one only executes jobs
    replicaCount: 2
persistence:
  enabled: false     # four pods cannot mount one ReadWriteOnce volume

An api node still creates jobs: starting an instance, completing a task and committing a variable all write to ACT_RU_JOB. It simply never acquires them.

"Runs no scheduled work" means cluster work. What separates the two is the lease: sixteen of the nineteen scheduled methods take a lease in job_leases before running, and on an api node that claim is denied immediately, without even reaching the database. The three that take no lease keep running on every node on purpose, because what they repair or publish is that node's state: delegate reconciliation, the AI queue's saturation sampling, and the node's membership heartbeat (ClusterMembershipService). ScheduledWorkIsLeasedOrPerNodeTest holds that list of three and fails a new @Scheduled method that does neither.

"Does not migrate" is the shared set at boot. Tenant-schema Flyway stays on in an api node — it is what creates a new tenant's schema and migrates it on that tenant's first request, and turning it off would hand a new tenant an empty schema and report success.

When to split. It is not about HTTP. The throughput that runs out is AI steps: one spends seconds to minutes blocked on the model holding a thread and a tenant connection, and that is the number adding workers grows. If the bottleneck is HTTP requests, backend.replicaCount already answers it.

Who migrates. The roles that execute jobs (all and worker) run the shared schema's Flyway set at boot; api never does. That is why backend.role: api without backend.worker.enabled: true is refused at render time: nothing would execute a job and nothing would migrate, and neither failure announces itself — the work simply accumulates unacquired. The reasoning, and the alternative that was rejected (a pre-upgrade Job), are in specs/core/job-worker-nodes.md → Who migrates.

Nothing orders the two Deployments

Kubernetes starts api and worker at the same time. An api pod can come up while the migration is still running; the right behaviour is for it to refuse readiness until the schema is the one it expects, and that half is not implemented yet in the application. Until it is, on an upgrade that carries a migration, install with --wait and check the worker's rollout before sending traffic.

Secrets

Leave secrets.jwtSecret and secrets.encryptionKey empty and the chart generates them, re-reading the existing Secret on upgrade so it never rotates by accident.

Keep the APP_ENCRYPTION_KEY

It decrypts credentials already stored in the database: the SSO secret, the SMTP password, the AI provider key (ai_models, rag_config), the webhook signing secrets and every tenant variable marked secret. Losing it does not fail immediately — it fails hours later, on the first integration that tries to read its own password. The Secret carries helm.sh/resource-policy: keep so helm uninstall does not take it along, and the installation notes print the backup command.

What is encrypted in the database

Third-party credentials are not stored in clear text. The columns below are encrypted with AES-256-GCM using APP_ENCRYPTION_KEY, on write and on read, transparently to the application:

Column	What it holds
ai_models.api_key, rag_config.api_key	The customer's credential with the AI provider
webhooks.secret	The key signing each delivery's HMAC
tenant_variables.value_text	Tenant variables, including the ones marked secret
sso_connections.client_secret, e-mail settings	SSO and mail-server credentials
ACT_RU_VARIABLE.TEXT_, ACT_HI_VARINST.TEXT_ (type encryptedSecret)	A password typed into a form field of type Password

At boot the application encrypts any value still in clear text and logs how many there were; the expected number is zero.

The last row is the only one that expires: a password typed into a form leaves the runtime tables when its instance ends, and a nightly sweep wipes the historic copy after FORM_PASSWORD_HISTORY_RETENTION_DAYS days (7 by default). It is the one credential with no home in the vault — it is born mid-process and belongs to a single instance.

A request body never reaches the log

The HTTP request log records from status 400 up and never writes the body that was sent, on any route. This depends on no field being marked and no path being listed: a form submission that fails carries the typed values in its body, and one of them may be a password. What stays in the log is the method, the path, the query (with password and token already obfuscated), the headers (with Authorization and X-API-Key already obfuscated), the status and the response body — which is where the validation error explaining the failure lives.

Credentials the system only checks are not encrypted but hashed, which is stronger: user password, API key, session token and e-mail token. The difference is whether the original value has to come back: the provider key has to be sent to the provider again, a password never does.

Encryption at rest protects the database, not the application

A backup dump, a read replica or direct database access do not reveal these credentials. A compromised process of the application itself does — it holds the key and decrypts by construction.

Recommended: SECRETS_INJECT_INTO_PROCESS_VARIABLES=false

With true (the default) a secret variable is copied into the instance's variables and Flowable stores it in clear text in the runtime and history tables. With false it never enters the instance — expressions use secrets.get(execution, 'key') instead, and no credential lands in the engine's tables. See Tenant Variables.

To manage secrets externally, use secrets.existingSecret with the keys: DB_PASSWORD, JWT_SECRET, APP_ENCRYPTION_KEY, S3_ACCESS_KEY, S3_SECRET_KEY.

GEMINI_API_KEY is optional — include it to configure a bootstrap AI model; without it the platform boots and AI features fail on the call, with a 409 and a sentence saying where to register a model. Do not include it empty: an empty variable is set, not absent, and Spring AI refuses to start against one. That is how the backend used to CrashLoopBackOff while helm install reported success.

External vault

By default every credential lives in the database, encrypted with APP_ENCRYPTION_KEY. An installation that already keeps credentials in HashiCorp Vault can connect the installation's vault: from then on a credential may be stored as a reference — vault://<mount>/<path>#<field> — and the platform reads the value from the vault at the moment of use, keeping no copy. With no vault connected nothing changes, and the installation starts as always.

Today the only supported vault is HashiCorp Vault (vault://). Azure Key Vault, AWS Secrets Manager and GCP Secret Manager are not supported yet.

secretBackend:
  vault:
    enabled: true
    uri: https://vault.company.internal:8200
    namespace: ""            # Vault Enterprise namespace, if any
    kvVersion: 2
    auth:
      method: kubernetes     # or approle
      kubernetes:
        role: flowia
        mountPath: kubernetes
        audience: vault
      approle:
        roleId: ""
        mountPath: approle
        secretIdSecretRef: { name: "", key: secret-id }
    tenantPathTemplate: "kv/flowi/tenants/{tenant}/"
    installationPathPrefix: "kv/flowi/platform/"
    cacheTtl: 5m

The platform authenticates with its own identity. With kubernetes, the pod presents a projected token of the chart's service account, with the audience audience; with approle, the roleId goes in the values and the secret-id is mounted as a file from the named Secret — it never becomes an environment variable. A static token is not accepted: the chart refuses to render it and the prod profile refuses to start with it. An http:// address is refused too, unless allowInsecure: true.

Each tenant only reads its own prefix. A tenant credential (secret variable, ${var.x} of a robot or an inbox, webhook secret) must point inside kv/flowi/tenants/<tenant-slug>/; an installation credential (AI model key, embedding key, SMTP password, SSO client secret) inside kv/flowi/platform/. The platform checks on save and on read. Mirror the same rule in the vault policy:

path "kv/data/flowi/tenants/*" {
  capabilities = ["read"]
}

path "kv/data/flowi/platform/*" {
  capabilities = ["read"]
}

vault auth enable kubernetes
vault write auth/kubernetes/config kubernetes_host="https://kubernetes.default.svc"
vault policy write flowia flowia.hcl
vault write auth/kubernetes/role/flowia \
  bound_service_account_names=<release>-flowia \
  bound_service_account_namespaces=<namespace> \
  audience=vault token_policies=flowia token_ttl=1h

The policy grants read only: the platform never writes to the vault. On KV version 1 the policy path has no data/.

An unreachable vault fails visibly. A value read is kept in memory, in the pod itself, for cacheTtl (5 minutes). Once that expires, if the vault does not answer the read fails — there is no stale copy in reserve: the AI step goes to retry and, once retries run out, to the failures panel; the webhook delivery is retried; the e-mail stays queued; the SSO login says the provider's secret is in an unreachable vault, and e-mail and password login keeps working. The state shows on /actuator/health/secretBackend as VAULT_UNREACHABLE, deliberately outside the /actuator/health aggregate: the chart's probes hit it, and a vault outage must not restart every pod.

Every read that reaches the vault is audited, without the value. The secret_read_audit table keeps when, which tenant, for what (AI model, webhook, variable…), which reference and which process step read it, and the outcome; secret_read_hourly counts every resolution per hour, cache hits included. Both delete what is older than 90 days.

AI provider

The default is Gemini through its OpenAI-compatible API, but any compatible endpoint works — vLLM, an internal gateway, Ollama behind a proxy:

ai:
  baseUrl: http://llm-gateway.internal:8000/v1/
  chatModel: qwen2.5-72b-instruct
  embeddingModel: bge-m3
  embeddingDimensions: 1024

embeddingDimensions sizes the pgvector column: changing it after documents are indexed requires reindexing the whole knowledge base.

OCR and knowledge base ingestion

The backend image ships Tesseract and the por, eng and spa languages (tessdata_fast) — about 47 MB more in the image, measured on 2026-10-19. OCR runs as a separate process inside the container: the document never leaves the installation and there is no token and no ai_usage_logs row. An image built without the binary works the same, only with OCR off — the upload screen says which one it is.

The cost is CPU: estimate 1 to 3 seconds per page at 300 DPI per core, and RAG_OCR_CONCURRENCY processes per node (1 by default). A large load of scanned PDFs competes with the requests of the same node; in an installation with node roles, leave ingestion to the worker.

The orphan window (RAG_ORPHAN_VECTOR_RETENTION_DAYS, 30 days) now also decides the cost of restoring an older document version: within it the restore reuses the vectors and spends no tokens; after it, the file is embedded again. Shortening the window saves disk and makes restoring more expensive.

The knowledge base restricted-access record (kb_retrieval_log, one row per search that returned or denied a classified excerpt) is pruned daily, under a lease, after RAG_RETRIEVAL_LOG_RETENTION_DAYS (365 days; minimum 1). It is an installation window: it does not appear on the tenant retention screen.

The sources of each AI step (ai_rag_retrievals, in the tenant schema) go with the instance, in the same batch as the Flowable history — so they are only pruned when history retention is on. The paragraph that asks the agent to cite the excerpt is written in the language of RAG_CITATIONS_INSTRUCTION_LOCALE (pt-BR by default; en or es), because the product has no per-tenant language.

SharePoint and OneDrive sync

Off by default: with KNOWLEDGE_SOURCE_PROVIDERS empty (app.knowledge.sources.providers) no screen offers the feature, the API refuses a connection and the job syncs nothing. To turn it on, set KNOWLEDGE_SOURCE_PROVIDERS=microsoft.

The hosts are fixed per cloud, not configurable and never typed by anyone — allow them in the firewall and in the outbound proxy:

Cloud	Hosts
GLOBAL	login.microsoftonline.com, graph.microsoft.com, *.sharepoint.com, *.files.1drv.com
US_GOV	login.microsoftonline.us, graph.microsoft.us, *.sharepoint.us
CHINA	login.chinacloudapi.cn, microsoftgraph.chinacloudapi.cn, *.sharepoint.cn

*.sharepoint.* and *.files.1drv.com are the download destinations (Graph redirects there, without the token). Any other host — even in a link Graph itself returns — is refused, and SsrfValidator still blocks internal addresses. Outbound proxy: the standard JVM properties (-Dhttps.proxyHost=… -Dhttps.proxyPort=… in JAVA_TOOL_OPTIONS) apply to these calls.

Variable	Default	What it does
KNOWLEDGE_SOURCE_PROVIDERS	empty	microsoft turns SharePoint and OneDrive on
KNOWLEDGE_SOURCE_MAX_ITEMS_PER_RUN	500	downloads per round before stopping at a page boundary and continuing the next minute
KNOWLEDGE_SOURCE_CONCURRENCY	2	sources syncing at once per node
KNOWLEDGE_SOURCE_RUN_RETENTION_DAYS	90	maximum age of kb_source_run rows (round history)
KNOWLEDGE_SOURCE_MAX_RETRY_ATTEMPTS	10	attempts of a failed download before the failure becomes permanent

The job (knowledge-source-sync, every minute) takes a lease: in a cluster with node roles it runs on the worker node only. The Entra app secret lives in a tenant secret variable (a value or a vault:// reference) and is read for each token; the token stays in memory only, on each node, until it expires. Shared migration V0141 creates knowledge_connection and knowledge_source, with each source's exclusive claim; tenant migration V0034 the item, round and failure tables, with the retry queue. A download has an overall 10-minute deadline and a Graph JSON response a 16 MB ceiling.

External knowledge bases

An external base queries a customer's Elasticsearch or OpenSearch on every step (Knowledge Bases). The network is opened by the operator, never by a tenant:

Variable / Helm value	What it does	Default
KNOWLEDGE_EXTERNAL_ALLOWED_INTERNAL_HOSTS / app.knowledge.external.allowedInternalHosts	internal network destinations an external base may query, as host and port (es.corp.internal:9200, 10.20.0.5:9200). A host without a port covers only the scheme's default port (443 for https). Outside this list an internal address is refused	empty
KNOWLEDGE_EXTERNAL_ALLOW_INSECURE_INTERNAL / app.knowledge.external.allowInsecureInternal	accepts http for a host of the list above. Every other destination needs https	false
KNOWLEDGE_EXTERNAL_EGRESS_DEFAULT_POLICY / app.knowledge.external.egressDefaultPolicy	deny refuses every external base of a tenant without allowed destinations; allow accepts any public https host when the tenant declares no list	deny
KNOWLEDGE_EXTERNAL_MAX_RESPONSE_BYTES	ceiling of the index answer; above it the query fails unread	1048576
KNOWLEDGE_EXTERNAL_MAX_CONCURRENT_CALLS	simultaneous external queries per node; one that finds no slot within its own time limit fails as unavailable	32
KNOWLEDGE_EXTERNAL_TEST_QUERIES_PER_MINUTE	how many times a minute one person may use Test a query	20

The cluster name is resolved once per query, every address is checked, and the connection goes to one of those checked addresses — the original name travels in Host and SNI, and the certificate is verified against it. A DNS answer that changes between the check and the connection cannot redirect the query into the internal network. This pinning covers external bases only: webhooks, the agent's HTTP tool and SSO check every address of the name and then let their HTTP client resolve it again to connect — the DNS-rebinding window stays open there, bounded by each one's destination list. The certificate is checked against the JVM trust store: a cluster with its own CA needs that CA in the trust store, like the vault. Redirects are never followed, and the whole answer, body included, must arrive within the base's time limit. The query count (knowledge_external_calls_hourly) is written per node every 5 s and pruned daily, under a lease, after 400 days. These queries make no model call.

Cluster with no internet

cd helm
make mirror REGISTRY=registry.corp.internal/flowia TAG=1.0.0
helm install flowia ./flowia -n flowia --create-namespace -f examples/airgapped.yaml

Nothing else reaches out: no remote subchart, no init container that downloads anything.

Operating it

cd helm
make status                  # release, pods, services, volumes
make logs                    # follows the backend
make smoke                   # helm test: backend health + SPA from inside the cluster
make upgrade VALUES=my.yaml

After installing, log in as SUPER_ADMIN and paste the installation's licence under Platform → Licence. Verification is offline: there is no activation server and no need for outbound internet access.

Chart tests

cd helm && make test

The assertions need only helm — no cluster, no plugin, no network. They render the chart under several sets of values and check object shape, database wiring, ingress routes, storage modes, secret handling and every guard that has to fail at render time rather than in the pod.

CI runs this on every push, plus kubeconform -strict against the real Kubernetes API schemas and helm package, publishing the .tgz as a build artifact.

Flyway — database migrations

Two sets, applied at different moments:

Path	Scope	When it runs
src/main/resources/db/migration/	The public (shared) schema	At startup, and only on the roles that execute jobs (all and worker) — an api node assumes the schema is already migrated
src/main/resources/db/tenant/	Each tenant's schema	At tenant provisioning; on that tenant's first request on each node; and on the background sweep, for a tenant with no traffic

No node migrates any tenant at startup

Since 2026-08-28 boot time no longer grows with the number of tenants. The consequence for whoever deploys is that a tenant schema's migration arrives with that tenant's first request after the release, or with the sweep — not with the pod coming up. See Multi-tenancy.

The tenant path is db/tenant/, not db/migration/tenant/

The shared Flyway points at classpath:db/migration and sweeps everything nested under it. A tenant migration placed in db/migration/tenant/ would also be executed against the public schema — the exact opposite of what it is for. See Multi-tenancy for the full mechanism.

Every migration must be readable by the revision before it

In production the new revision boots with no traffic, goes through the smoke and only then receives users (see Post-deploy smoke). It runs Flyway when it boots, against the shared database: if the smoke fails, the database has already been migrated and the previous revision keeps serving on it. So a breaking schema change ships in two releases — first expand (new column, both written), then contract (drop the old one). A DROP or RENAME of something the previous release still reads turns a failed smoke into an outage. The full table is in skills/backend/flyway-migration.md.

The document versions migration rewrites vector_store

V0136 (global bases, in public) and V0028 (each tenant) create the document and version tables and update the metadata of every knowledge base chunk already indexed, to say which document and version it belongs to. It is one UPDATE over the whole vector_store in a transaction: while it runs, ingestion and search in that schema wait for the lock. A tenant with hundreds of thousands of chunks may take minutes — rehearse on a copy (below) and plan the window.

Rehearse the migration on a copy before shipping it

A migration that only creates a table needs no rehearsal. One that rewrites existing rows does: how many rows it touches — and which ones it cannot convert — only shows up against real data. That was the case for V0097 (the definitions' tenant_id_ moves from the UUID to the slug, across four Flowable tables) and V0098 (robots.parameters becomes jsonb, and anything that is not a JSON object is nulled).

./deploy/rehearse-migration.sh --dump backup.sql

The script starts a throwaway Postgres, restores the dump, prints the state before, applies the pending migrations and prints the state after. The container is removed at the end, with or without an error, and the dump is never written to. It takes pg_dump in plain or custom format, and --migration V0097 limits which ones run.

The report answers three questions: how many rows each table moves, how many match no tenant at all and therefore stay as they are, and how many robots would lose their parameters for not being a JSON object. That last number is the one worth a conversation before the deploy: those robots will need reconfiguring from the screen.

Upgrading to hybrid search: V0031 holds writes to vector_store

The tenant migration V0031 (and V0139, for public) creates the keyword index and fills it from the chunks that already exist, inside the migration itself. While the backfill runs, the migration's transaction holds that tenant's vector_store: ingestions and any other write to it wait, while reads go on. The measured cost is about 22 s per 134 thousand chunks (≈3.3 thousand characters each); a small tenant will not notice. Ship the version outside heavy ingestion hours, or rehearse with rehearse-migration.sh to learn the time for your largest tenant. No token is spent and no vector is rewritten.

Script definitions: refused on deploy, and what to do about the old ones

What the platform guarantees, and what it does not. The JavaScript engine (nashorn-core) was removed from the core, and deploy, publish, global-template import and DMN deployment answer 422 to: scriptTask, a script element or script / scriptFormat attribute, type="script" on listeners and CMMN tasks, a language= other than juel, DMN with a script expressionLanguage, any flowable:type / activiti:type outside dmn, java and case (so shell, http, mail, camel and mule), a class= that is not a platform class, web services (implementation="##WebService", operationRef, <interface>, <operation>, WSDL <import>), and ${...} expressions that mention reflection, the engine, jdbcTemplate or dataSource.

This is not "the platform never evaluates code": it evaluates ${...} expressions, and what protects it is a sandbox inside the engine, which also covers definitions already stored and anything the deploy check cannot read ('a' + 'b'). The sandbox: (1) no expression calls methods or reads properties of Class, ClassLoader, java.lang.reflect.*, java.lang.invoke.*, Runtime, Process, ProcessBuilder, Thread, System, nor getClass on any object, nor the engine services (getProcessEngineServices, getCommandContext...); (2) expressions reach only the secrets and requiredVariables beans and the beans that are delegates or listeners (JavaDelegate, ExecutionListener, TaskListener, CMMN delegates and listeners), never jdbcTemplate, dataSource or repositories; (3) the engine refuses, when it reads the XML, script tasks, step types outside the list, script listeners and class= outside the platform. It applies to the process, case and decision engines. What stays possible for whoever may deploy: ${...} with execution.setVariable, conditions on variables, delegateExpression to platform delegates and listeners. A modeler can not run arbitrary code on the server this way; delegates uploaded as JARs, which run in the JVM, remain an act of whoever administers the platform.

Definitions already deployed with a script stay in the database. Without the engine, an instance that reaches the script step fails with Can't find scripting engine for 'javascript', and publishing a new version that still has a script is refused. To list what needs fixing, run this on the platform database (the ACT_* tables live in the public schema):

SELECT 'BPMN' AS kind, d.tenant_id_, d.key_, d.version_, d.id_ AS definition_id
  FROM act_re_procdef d
  JOIN act_ge_bytearray b ON b.deployment_id_ = d.deployment_id_ AND b.name_ = d.resource_name_
 WHERE convert_from(b.bytes_, 'UTF8')
       ~* '(<([a-z0-9_]+:)?scriptTask[ >/])|scriptFormat=|(<([a-z0-9_]+:)?script[ >])|type="script"'
UNION ALL
SELECT 'CMMN', d.tenant_id_, d.key_, d.version_, d.id_
  FROM act_cmmn_casedef d
  JOIN act_cmmn_deployment_resource r ON r.deployment_id_ = d.deployment_id_ AND r.name_ = d.resource_name_
 WHERE convert_from(r.resource_bytes_, 'UTF8')
       ~* '(<([a-z0-9_]+:)?script[ >])|scriptFormat=|type="script"'
UNION ALL
SELECT 'DMN', d.tenant_id_, d.key_, d.version_, d.id_
  FROM act_dmn_decision d
  JOIN act_dmn_deployment_resource r ON r.deployment_id_ = d.deployment_id_ AND r.name_ = d.resource_name_
 WHERE convert_from(r.resource_bytes_, 'UTF8') ~* 'expressionLanguage="(groovy|javascript|js)"'
ORDER BY 1, 2, 3, 4;

For each row, open the process in the modeler, replace the script step with a service task using an expression, a delegate or a DMN decision, then deploy and publish the new version. The old version only stops running once a new version is published in its place.

Initial Super Admin

The initial user is created by the V0001__init.sql migration, which runs by itself on first startup — there is no manual step to perform.

Field	Value
Email	admin@flowiagentic.local
Password	Admin@1234
Role	SUPER_ADMIN

The email has no hyphen (flowiagentic, not flow-iagentic) — the database name does, which is where the confusion comes from. Logging in with the hyphenated version returns invalid credentials.

Change it immediately in production

The default password is public knowledge and must be changed before the application is exposed.

Production build (Frontend)

cd frontend
npm run build

The static files land in frontend/dist/ and can be served by any web server (nginx, Apache, CDN).

CI/CD

CI runs on GitHub Actions, with the workflows in .github/workflows/. There is no azure-pipelines.yml in the repository.

Workflow	What it does
ci.yml	Validation on every push to master/main and on every pull request
deploy-gcp-backend.yml	Deploys the backend to GCP
security.yml	Semgrep and OSV on every push; Dependency-Check and SBOM twice a week
deploy-gcp-frontend.yml	Deploys the frontend to GCP
deploy-gcp-docs.yml	Deploys this documentation
deploy-gcp-landing.yml	Deploys the marketing site
production-smoke.yml	Hourly: the smoke against production, and a check that what is serving has not fallen behind master

ci.yml has two parallel jobs:

1. Backend (Java 21) — brings up a pgvector/pgvector:pg16 Postgres as a service and runs mvn verify, excluding S3StorageProviderTest and AttachmentServiceTest, which depend on external storage. mvn verify does not start the application and does not write openapi.json: that artifact lives in the openapi profile, described in REST API.
2. Frontend (TypeScript) — npm run build, npm run test:run, npm run i18n:check, npm run i18n:literals and npm run lint:check. The type step is the full production build, not tsc --noEmit: on its own it skips the project references and lets through errors the production image catches.

i18n:check is blocking: one new string missing a translation in pt-BR, en or es fails the build.

i18n:literals covers the hole the previous step cannot reach. i18n:check compares keys between locales; a string that never went through t() is not a key at all, so there is nothing to compare and it crosses the step untouched — that is how the same defect was fixed once and came back. i18n:literals parses every .tsx with the TypeScript parser and counts text literals in the four props that reach the screen: label, placeholder, title and aria-label. A key, a variable or a template does not count; only a bare literal does. The ceiling per prop is frozen in frontend/scripts/ui-literals-baseline.json, under the same contract as the lint one: old debt passes, one new literal fails, and the number only goes down — npm run i18n:literals -- --update-baseline locks the gain in after a round of translation. Exceptions are never silent: test files are dropped by EXCLUDED_PATHS, and a specific site escapes only when declared in DELIBERATELY_LITERAL, with a written reason, at the top of scripts/check-ui-literals.mjs.

lint:check blocks differently. ESLint carries old debt (368 findings), and failing every build over it would only teach people to skip the step; so the count per rule is frozen in frontend/scripts/lint-baseline.json. Old debt passes; a rule that grows fails; and a rule nobody was violating has a ceiling of zero, so a new category of problem cannot arrive quietly. Fixed some findings? npm run lint:check -- --update-baseline locks the gain in — the command refuses to raise a ceiling, only to lower it.

Post-deploy smoke

CI proves the code; the smoke proves the installation. The chart test runs before the deploy, against MinIO and with no embedding provider — by construction it cannot see the production bucket, key or configuration. On 2026-10-01 that let through an upload that had answered 403 from GCS for four weeks.

How a deploy works now, for all three services (backend, frontend, docs):

1. the new revision boots on Cloud Run with no traffic, tagged candidate, with its own URL;
2. the smoke runs against that URL;
3. passed: update-traffic --to-latest moves users to it. Failed: traffic stays on the previous revision, the tag is removed and the workflow turns red.

The backend smoke is deploy/smoke/post-deploy-smoke.sh, and each step proves one piece of the environment:

#	Step	What it proves
1	GET /actuator/health answers UP	The process booted and reaches the database
2	Login and select-tenant on tenant smoke	JWT_SECRET and the auth chain
3	Upload an attachment, download it, compare the bytes	The real bucket, with the real credentials
4	Upload a .md to the knowledge base and wait for a new chunk (up to 60 s)	RAG configuration, embedding provider, key and ingestion
5	Start smoke_ping, claim and complete its task, check the instance ended	Flowable on the tenant and the catalogue
6	Delete the attachment and the document	The smoke tenant does not grow

Frontend and docs go through deploy/smoke/static-site-smoke.sh: GET / answers 200 and so does every /assets/* referenced by index.html.

The smoke makes no chat model call. Step 4 embeds a text of about fifty words and is metered as RAG_EMBEDDING of tenant smoke on the governance screen.

When the smoke is red

• On a deploy: production did not change — users stay on the previous revision. The FAIL line in the log names the step and carries the response body. Step 3 is storage (bucket, HMAC credentials); step 4 is RAG (Admin → AI settings → Vector store, and the RagIngestionAsyncService log).
• On the hourly run: production is broken without a deploy — revoked key, exhausted quota, changed bucket permissions. GitHub notifies the account that last changed the cron.
• On the drift check: a service is serving a commit more than 24 hours behind a master commit that touched its files. It is almost always a red CI blocking the deploy — look for the ci-red-on-master issue, opened by ci.yml when master fails and closed by the next green run.

Running it by hand

SMOKE_EMAIL=... SMOKE_PASSWORD=... ./deploy/smoke/post-deploy-smoke.sh https://app.flowiagentic.com

Against app.flowiagentic.com, /actuator lands on the SPA; pass the backend health in SMOKE_HEALTH_URL (the *.run.app URL of the flowia-backend service + /actuator/health).

Preparing an installation

Once per environment, with the super admin:

SUPER_ADMIN_EMAIL=... SUPER_ADMIN_PASSWORD=... SMOKE_EMAIL=... SMOKE_PASSWORD=... \
  ./deploy/smoke/bootstrap-smoke-tenant.sh https://app.flowiagentic.com

It creates tenant smoke, the smoke user as MODELER of that tenant only, and the smoke_ping process. It can run again without duplicating anything. The same SMOKE_EMAIL/SMOKE_PASSWORD pair goes into the GitHub Actions secrets.

The smoke itself is tested before it guards production: the Chart installed in a real cluster job runs deploy/smoke/ci-cluster-check.sh, which points RAG at a fake embedding provider, requires the smoke to pass, then runs it again with MinIO switched off and requires it to fail at step 3.