Architecture

Overview

┌──────────────────────────────────────────────────┐
│              Frontend (React 19 + Vite)          │
│   TanStack Router → Components → Axios → API     │
└────────────────────┬─────────────────────────────┘
                     │ HTTPS / JWT / WebSocket
                     ▼
┌──────────────────────────────────────────────────┐
│           Backend (Spring Boot 3.5)              │
│                                                  │
│  Security Filters                                │
│  ApiKey → RobotRpc → TenantFilter → JWT          │
│                    ↓                             │
│  Controllers (REST, OpenAPI 3.1)                 │
│                    ↓                             │
│  Services (@Transactional)                       │
│                    ↓                             │
│  Repositories (JPA) + Flowable Engines           │
│  (BPMN, CMMN, DMN, Task, History)                │
│                                                  │
│  + NotificationService (WebSocket STOMP)         │
│  + RobotExecutor (Scheduler + runner assignment) │
│  + WebhookService (CRUD + HMAC-SHA256; see note) │
│  + AttachmentStorage (S3; filesystem in dev only)│
│  + AgenticLlmCoreService (Spring AI + RAG)       │
└────────────────────┬─────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
  ┌──────────┐ ┌──────────┐ ┌──────────┐
  │  public  │ │tenant_abc│ │tenant_xyz│
  │ (shared) │ │(isolated)│ │(isolated)│
  └──────────┘ └──────────┘ └──────────┘
        └────────────┴────────────┘
                     │
              ┌──────┴──────┐
              │ PostgreSQL 16│
              │  + pgvector  │
              └─────────────┘

A tenant's schema does not hold everything that belongs to the tenant. Flowable's 53 ACT_* tables — definitions, instances, tasks, jobs and history — live in public and are isolated by the tenant_id_ discriminator rather than by the schema, and the same goes for the discriminator tables (ai_agents, robots, api_keys, …). Which tables live where is in Multi-tenancy.

Application layers

Backend

Controller   — REST endpoint, OpenAPI annotations, no business logic
     ↓
Service      — business logic, @Transactional, validation
     ↓
Repository   — JPA interfaces (Spring Data)
Engine       — Flowable BPMN/CMMN/DMN (through Services)
     ↓
Entity       — JPA models mapped to the database

Rules:

• A Controller never touches a Repository directly
• Services own every transaction
• Every interaction with Flowable goes through a Service

Frontend

Routes (TanStack Router)
     ↓
Pages (page components)
     ↓
Hooks (React Query — remote data) + Stores (Zustand — local state)
     ↓
lib/axios.ts (HTTP client with JWT + refresh interceptors)

On-demand loading

Every route is its own chunk: TanStackRouterVite runs with autoCodeSplitting, so a page component is downloaded only when you navigate to it. The login screen loads the application shell and nothing else — no BPMN/CMMN/DMN modeler, no diagram viewers, no form builder, no copilot. While a route chunk is in flight the router shows a loading indicator, and a link under the cursor is already being fetched before the click.

Two practical consequences for anyone touching the code:

• A route file exports Route and nothing else. Any other runtime export pins the whole module — and everything it imports — into the first load. A helper shared by two routes belongs in lib/ or -components/, never in a sibling route. Type-only exports are fine: types do not survive compilation.
• Diagram viewers always come from @/components/LazyDiagramViewers. Importing BpmnViewer, CmmnViewer or DmnViewer straight from the component drags bpmn-js/cmmn-js/dmn-js into everyone's first load, including people who only opened the login page.

npm run build ends by measuring everything dist/index.html references and fails if the total goes past the ceiling in frontend/scripts/bundle-budget.mjs. The same ceiling is re-asserted by src/__tests__/bundle-budget.test.ts. Raising it is a deliberate act: change the number and say why in specs/frontend/architecture.md.

Security filters (Backend)

The filters run as a chain on every request, in this order — it is registered in config/SecurityConfig.java and the order matters: TenantFilter runs before JwtAuthFilter, so the tenant is already in TenantContext by the time the user is authenticated.

Order	Filter	Responsibility
1	McpBasicAuthFilter	Authenticates the MCP server (/api/a/mcp) via HTTP Basic
2	ApiKeyAuthFilter	Authenticates requests carrying the X-API-Key header
3	RobotRpcFilter	Authenticates the robot sandbox RPC channel (/api/internal/robot-rpc/**)
4	TenantFilter	Reads the tenant from the signed JWT claim and populates TenantContext (ThreadLocal). X-Tenant-ID does not choose a tenant
5	JwtAuthFilter	Validates the Bearer JWT and populates the SecurityContext

There is no AuditLogFilter. What feeds the logs is the MDC, assembled inside the two filters above: TenantFilter writes requestId and tenantId, JwtAuthFilter writes userId. Looking for a dedicated auditing class is wasted time — correlating requests means grepping those three fields in the log lines.

Special components

TenantAwareDataSource

An AbstractRoutingDataSource that routes every database connection to the right schema based on TenantContext. There is one HikariCP pool per node and no pool per tenant: the search_path is applied when a connection is borrowed and the session is reset when it is returned, so the connection budget does not grow with the number of customers. The per-tenant pool was removed on 2026-09-09 — see Multi-tenancy.

RobotExecutorService and RobotSchedulerService

RobotSchedulerService fires robots on their cron expression, through ScheduledTaskRegistrar. RobotExecutorService records the execution as PENDING and hands it to RunnerAssignmentService, which picks a registered runner. The application never opens a robot JAR: the machine registered as a runner is what starts the sandbox and executes the code, and that is what keeps one customer's code off the disk that holds everyone else's configuration, credentials and attachments. The sandbox talks back to the platform over the RPC channel at /api/internal/robot-rpc/**.

WebhookService

Registers webhooks per tenant, delivers the payload over POST and signs every body with HMAC-SHA256 — every webhook has a secret: generated when left blank, returned only on create and on regenerate (POST /api/a/webhooks/{id}/secret) — the signature goes in the X-FlowIa-Signature header (sha256=...) and the event name in X-FlowIa-Event.

WebhookEventListener subscribes to the engine and maps five events to the documented names: instance.started, instance.completed, instance.failed, task.created and task.completed.

Before anything is sent, WebhookEgressGuard checks the destination host against the list the tenant declared (tenants.webhook_allowed_hosts), through the same EgressAllowList the AI agents' HTTP call uses. A destination outside the list is refused when the webhook is saved, with the host in the message, and a delivery already queued ends as BLOCKED — a terminal status, distinct from FAILED: FAILED is the destination saying no, BLOCKED is the platform. SsrfValidator runs after the list, so declaring localhost does not open an internal target. app.webhooks.egress.default-policy decides what an empty list means — allow keeps an existing installation working, deny is what a new installation should run. See specs/automation/webhook-egress-allowlist.md.

task.overdue does not exist

A missed due date is not an engine event — nothing happens at the instant the date passes. Delivering that event would take a periodic sweep over open tasks, with state to avoid re-sending on every pass. It is not implemented, and saying so is better than listing it beside the other five.

The listener only records the delivery; the job is what sends it. The HTTP call does not happen inside the process transaction, and that is deliberate: a completing instance would hold a database connection for the duration of a call to another company's server, timeout included. It was the same failure shape as an AI provider call inside the transaction, which serialised an entire tenant and, under concurrency, exhausted the pool — and which left the transaction on 2026-09-09.

The practical consequence is that delivery is not immediate: one minute in the worst case. And if the process transaction rolls back, the delivery rolls back with it — an instance that never started should not announce that it did.

Retry is not exponential. A @Scheduled job runs every 60 seconds, sweeps deliveries in PENDING and RETRYING and re-sends each one. The interval is always that same minute, and on reaching 3 attempts the delivery is marked FAILED and never picked up again.

AI step concurrency

The AI step left the engine's thread. Since 2026-09-09 on BPMN and 2026-09-10 on CMMN it writes a reservation, hands the thread and the transaction back, and the model call happens on a pool of its own (aiStepTaskExecutor, in AiStepExecutorConfig). When the answer arrives, completion goes back through the engine — triggerAsync on BPMN, the plan item's trigger on CMMN. While the provider thinks, the step occupies no Flowable job thread and holds no transaction or database connection.

The sustained ceiling on concurrent AI steps is now the core-pool-size of that pool: 8 per installation (AI_STEP_CORE_SIZE), with a queue of 100 (AI_STEP_QUEUE_CAPACITY) and a maximum provider wait of 120 s (AI_STEP_PROVIDER_TIMEOUT_SECONDS). A full queue does not park the instance in silence: the reservation is closed as a failure and the step shows up for reprocessing. A reservation's lease is twice the provider timeout, and a periodic reaper redoes the step whose node went down.

Between the pool and the provider sits the model limiter (AiModelCallLimiter, one gate per model and provider key, wired in AiClientFactory). It spaces requests according to app.ai.rate-limit.requests-per-minute, caps concurrent calls, counts every tool round and pauses the model when the provider answers a rate-limit 429. Calls with a person on screen (copilots, playground) go first without waiting. A step that gets no slot goes back to the queue without calling the provider — it spends no attempt and writes no usage row. The limit is per node. A rate-limit 429 is retried after the wait the provider asked for, within the step's budget; an exhausted-quota 429 is final. Details in specs/ai/ai-step-execution.md, section Rate limit.

As throughput — the form a commercial conversation needs:

steps per minute = AI pool core-pool-size × 60 ÷ call latency in seconds

With today's default (8) and an AI call of roughly 20 s, the arithmetic gives 24 steps per minute, 1,440 per hour, per installation. That is arithmetic, not a measurement: the 2026-09-03 rig that produced the published numbers ran against the old, 2-slot design and is in specs/core/flowable-job-pool.md. The new shape, for processes and cases, is in specs/ai/ai-step-execution.md. Measure with your own load before promising throughput to anyone.

The Flowable job pool is still a single ThreadPoolTaskExecutor bean (applicationTaskExecutor), defined in FlowableTaskExecutorConfig and shared by both engines, with core-pool-size 2 (FLOWABLE_EXECUTOR_CORE_SIZE) and max-pool-size 3. It is no longer the AI ceiling, but it is still the ceiling on everything else — timers, async continuations, and the job that completes the AI step once the answer is in.

Raising the number of job threads on a single node is not free capacity: a job can hold two connections from the shared pool (spring.datasource.hikari.maximum-pool-size, 10 by default) because of the REQUIRES_NEW around quota deduction. The floor is exactly max-pool-size × 2 + 1 = 7, and it applies to app.tenant.pool.maximum-pool-size (7) as well, which is one tenant's ceiling inside that pool. Raising FLOWABLE_EXECUTOR_MAX_SIZE forces both up with it, on pain of the tenant no longer finishing steps. Where the number actually comes from and the configurable full-queue behaviour (queue-full-policy, ABORT or CALLER_RUNS) are documented in specs/core/flowable-job-pool.md and specs/core/job-worker-nodes.md.

A process deployed before the change stays on the old path until its next version

On BPMN the step only reserves and returns when the deployed XML carries flowable:triggerable="true". The modeller writes the attribute when the implementation is set to ✨ AI Agent, and deploy adds it to every flowable:delegateExpression="${aiAgentDelegate}" step that arrives without it — from the modeller, Import XML, the API or a global template being distributed. Deploying a new version of the process is enough, even with nothing changed in the diagram.

What does not change by itself: a version deployed before this rule keeps calling the provider on whichever thread reached it for as long as it is the published one, and an instance already running follows the version it started on. A hand-written flowable:triggerable="false" is kept. In a case (CMMN) there is no attribute to write: the behaviour comes from the delegate class itself and applies to every case as soon as the version is up.

A job the engine abandons

When a job's failure is the absence of a database connection, Flowable's own failure handler fails for the same reason as the work it was about to handle: the exception escapes the executor and the row in act_ru_job keeps the lock the acquisition thread wrote. That used to be invisible — no log naming the job, no dead-letter row, and the only way to find out was to query act_ru_job by hand.

That escape is now caught at the frame that still knows which job it is, and the job goes back to the queue with one retry less, carrying the reason. On its last attempt it lands in the dead-letter queue with that same reason — so it shows up in GET /a/instances/failed-jobs and in the per-step monitor (JOB_RETRYING while retries remain, JOB_FAILED afterwards), like any other failure. An ERROR log line names the job, tenant, handler and instance, and the counter flowi.job.abandoned (tags tenant, handler, outcome) moves even when the database is the thing that is down — that is what an alert should watch. outcome=stuck means even the requeue could not be written: the job then returns only when its lock_exp_time_ expires, and the log line is the only evidence it exists until then. Details in specs/core/abandoned-job-recovery.md.

Coordination between instances (job lease)

With more than one instance of the application running against the same database, every @Scheduled method would fire on every node — the webhook retry above would deliver the same payload to the customer once per node, and a robot on a CRON would run twice. A database lease prevents that (table job_leases, shared schema): each task competes for ownership of a fixed name for a short period before running, only the node that already owns it renews, and an expired lease — a node that died — is taken over by another on the next attempt. It covers the cleanup and retry @Scheduled methods plus the per-robot CRON trigger in RobotSchedulerService.

The login rate limiter used to be the exception, counting requests in its own node's heap. It stopped being one on 2026-08-28: it and each API key's budget are counted in a shared table (rate_limit_counters), because a per-node limit hands out N times the number the screen promises the moment someone runs a second replica. Two other things that lived in one pod's heap left with it — real-time events and tenant migration at boot — and are described in specs/core/cluster-fanout.md.

The exception that remains is deliberate and the other way round: delegate reconciliation runs without a lease, on every node, because what it repairs is per-node state (specs/core/delegate-convergence.md).

Node role (APP_NODE_ROLE)

The lease above is also where the node's role is enforced. APP_NODE_ROLE picks between all (the default, which does everything, as it always has), api and worker:

Role	HTTP	Acquires Flowable jobs	Cluster scheduled work and robot CRONs	Shared Flyway at boot	Default tenant ceiling
all	everything	yes	on the elected node	yes	7
api	everything	no	no	no	3
worker	/actuator only	yes	on the elected node	yes	7

An api node still creates jobs — starting an instance, completing a task and committing a variable all write to ACT_RU_JOB — it simply never acquires one. And "runs no scheduled work" means cluster work: on an api node JobLeaseService#tryAcquire denies the lease immediately, without reaching the database, which turns every lease-taking scheduled task into a no-op. The three that take no lease keep running there on purpose, because what they repair or publish is that node's own state — delegate reconciliation, the AI queue's saturation sampling, and the node's membership heartbeat. ScheduledWorkIsLeasedOrPerNodeTest fails a new @Scheduled method that does neither.

The role is a single value, and everything else it decides is a default: an explicit FLOWABLE_JOBS_ENABLED or TENANT_POOL_MAX_SIZE still wins over it. A misspelled role refuses to boot rather than becoming all and serving under a role other than the configured one.

The connection ceiling, printed at boot

NodeConnectionBudget computes, on every boot, what the installation asks of the database: (shared + 1) × expected pods. The parenthesis is what one pod costs — each pod is its own JVM with its own pool — and the +1 is the cluster bus connection. The line is always printed, with the node's role and the decomposition (... = N per pod × M pods = T) rather than only the total; past APP_DATABASE_MAX_CONNECTIONS it becomes a warning with the overshoot and the levers that remain, and with APP_DATABASE_BUDGET_ENFORCE=true the node refuses to boot. The number is worst case, not current consumption.

There is no tenant term. It left with the per-tenant pool on 2026-09-09, so the next customer does not move this number and what crosses the ceiling is the pod. APP_DATABASE_EXPECTED_TENANTS is still printed next to the sum, precisely to show that it does not enter it. With APP_NODE_AFFINITY_ENABLED=true the line says that affinity buys no connection at all — there is no per-tenant term left for it to stop multiplying — and that what it still buys is job ownership and plan locality. See specs/core/job-worker-nodes.md, specs/core/connection-pool-budget.md and specs/core/tenant-node-affinity.md.

NotificationService

Uses Spring WebSocket (STOMP) to push real-time notifications to the frontend. Each user subscribes to a personal channel.

The broker is in memory, but nothing publishes to it directly: every message goes through RealtimeMessagePublisher, which sends it over Postgres LISTEN/NOTIFY, and each node delivers to its own sessions. That is what makes an event published on one pod reach a browser attached to another, with no external broker and no session affinity. See specs/core/cluster-fanout.md.

FormVersionLockListener

A Flowable TASK_CREATED listener. When the engine creates a task, this listener:

1. Reads the formKey declared in the BPMN/CMMN model
2. Calls FormService.resolvePublishedVersion(formKey) to get the published version (or the latest one if nothing has been published)
3. Stores the version UUID in the task-local variable _form_version_id

This guarantees the task always renders the form schema that was in force when it was created. isFailOnException = false — a failure to lock the version does not block task creation.

AiClientFactory and dynamic PgVector

Flowi Agentic integrates Spring AI natively. Because the platform lets you change the provider and the embedding model dimensions from the database (table rag_global_config), the vector database (PgVectorStore with HNSW indexing on PostgreSQL) is rebuilt on the fly.

AiClientFactory reads the configuration and instantiates the VectorStore dynamically on demand. When the dimension changes structurally, the database drops and rebuilds the vector_store table, invalidating older vectors that then have to be regenerated through the reindex endpoint.

The factory is also the only place that assembles a model client: it is where the usage-tracking advisor is attached to the ChatClient and where the embedding model gets its metering. A client assembled anywhere else produces a call that burns tokens and never shows up on the tenant's report. EveryModelCallIsMeteredTest fails such an assembly, and fails a model invocation whose method does not open and clear the usage context.