Reference

FlowDSL Sync

How FlowDSL diagrams stay in sync with Go module code — from Studio edit to deployed container.

FlowDSL Sync Architecture

How FlowDSL diagrams stay in sync with Go module code — from Studio edit to deployed container.

Status

Draft — proposed architecture, not yet implemented.


The Problem

FlowDSL .flowdsl.yaml files in backend/flows/ currently describe business flows as documentation only. The runtime doesn't read them. There's no enforcement that Go handlers implement all declared nodes, and changing an edge's delivery mode requires manual code changes. This document defines an architecture where the diagram is the authoritative source of wiring.


Core Assumptions

  • Redeploy is acceptable. A container restart when a flow changes is fine. Hot-reload is a later optimization, not a requirement.
  • Compile-time safety over runtime safety. If a node in the diagram has no handler, the build should fail — not the server at startup.
  • Modules own their nodes. Each Go module declares which FlowDSL operationId values it implements. The framework discovers them automatically.
  • Delivery mode is diagram-owned. Changing direct → durable on an edge should require zero Go code changes — only the YAML changes.

Architecture Overview

text
┌──────────────────────────────────────────────────────────────────────┐
│  1. DIAGRAM LAYER  (Studio / .flowdsl.yaml)                          │
│                                                                      │
│  backend/flows/user_registration.flowdsl.yaml                        │
│  backend/flows/module_submission.flowdsl.yaml                        │
│  backend/flows/email_notification.flowdsl.yaml                       │
│  ...                                                                 │
│                                                                      │
│  Source of truth for: nodes, edges, delivery modes, retry policies,  │
│  idempotency keys, packet types.                                     │
└──────────────────────┬───────────────────────────────────────────────┘
                       │
                       │  go run ./cmd/generate/ flows
                       │  (runs in CI after any .flowdsl.yaml change)
                       ▼
┌──────────────────────────────────────────────────────────────────────┐
│  2. CODEGEN LAYER  (backend/flows/gen/)                              │
│                                                                      │
│  user_registration_flow_gen.go                                       │
│  module_submission_flow_gen.go                                       │
│  email_notification_flow_gen.go                                      │
│                                                                      │
│  Generated per flow:                                                 │
│  - Typed XxxFlowNodes struct (one field per node operationId)        │
│  - RegisterXxxFlow(bus, nodes) wiring function                       │
│  - Edge wiring is compiled —  delivery mode → go-events call         │
│  - Checked into git (diff = what the diagram change produced)        │
└──────────────────────┬───────────────────────────────────────────────┘
                       │
                       │  go build ./...  (fails if nodes unimplemented)
                       ▼
┌──────────────────────────────────────────────────────────────────────┐
│  3. NODE HANDLER LAYER  (backend/modules/*/flows.go)                 │
│                                                                      │
│  community/flows.go   → func (m *Module) UserRegistrationNodes()     │
│  marketplace/flows.go → func (m *Module) ModuleSubmissionNodes()     │
│  email/flows.go       → func (m *Module) EmailNotificationNodes()    │
│                                                                      │
│  Developer writes: pure business logic functions.                    │
│  No transport knowledge. Same service layer as HTTP handlers.        │
│  Scaffold generates stubs for new operationIds automatically.        │
└──────────────────────┬───────────────────────────────────────────────┘
                       │
                       │  compiled into binary
                       ▼
┌──────────────────────────────────────────────────────────────────────┐
│  4. RUNTIME  (started in cmd/api/main.go)                            │
│                                                                      │
│  flows.RegisterAll(bus, modules)                                     │
│  → starts goroutines for durable/ephemeral/stream edges              │
│  → direct edges become inline function calls                         │
│  → wiring is compiled in, not dynamic                                │
└──────────────────────────────────────────────────────────────────────┘

What Gets Generated

For each .flowdsl.yaml, the generator produces one *_flow_gen.go file.

Example: user_registration_flow_gen.go

go
// Code generated from user_registration.flowdsl.yaml. DO NOT EDIT.
// Source: backend/flows/user_registration.flowdsl.yaml
// Generated: 2026-04-10T11:30:00Z
package flows

import (
    "context"
    "time"
    "github.com/redelay/go-framework/flowdsl"
)

// UserRegistrationFlowNodes holds all node handlers for the user_registration flow.
// Every field must be set — the compiler enforces it at construction time.
type UserRegistrationFlowNodes struct {
    // source: receives POST /register
    ReceiveRegistration flowdsl.SourceHandler

    // transform: validates email, password, account_type
    ValidateRegistrationInput flowdsl.RouterHandler // valid → continue, invalid → reject

    // router: routes by account_type (developer / community / customer)
    RouteRegistrationByType flowdsl.RouterHandler

    // action: creates core user + community profile in MongoDB
    CreateUserAccount flowdsl.NodeHandler

    // action: queues verification email via email.send event
    SendVerificationEmail flowdsl.NodeHandler

    // publish: emits users.registered to Kafka
    PublishUserRegistered flowdsl.PublishHandler

    // terminal: returns 400 with validation errors
    RejectInvalidRegistration flowdsl.TerminalHandler
}

// RegisterUserRegistrationFlow wires the user_registration flow using the provided handlers.
// Edge delivery modes are compiled in from the diagram — changing the YAML regenerates this function.
func RegisterUserRegistrationFlow(bus flowdsl.EventBus, nodes UserRegistrationFlowNodes) {
    // Edge: RegistrationSubmitted → ValidateRegistration [direct]
    // Delivery: in-process, microsecond latency, no durability
    // (triggered externally by the HTTP handler calling nodes.ReceiveRegistration)

    // Edge: ValidateRegistration.valid → RouteByAccountType [direct]
    // Edge: ValidateRegistration.invalid → RejectInvalidRegistration [direct]
    // These are handled by the RouterHandler return value (port name)

    // Edge: RouteByAccountType.* → CreateUserAccount [durable]
    // Delivery: MongoDB-backed packet, retry x3, exponential backoff 2s
    // IdempotencyKey: {{payload.email}}-register
    bus.Subscribe(flowdsl.DurableConsumer{
        Topic:          "flow.user_registration.create_account",
        GroupID:        "flow.user_registration",
        IdempotencyKey: func(p map[string]any) string {
            return fmt.Sprintf("%v-register", p["email"])
        },
        RetryPolicy: flowdsl.RetryPolicy{
            MaxAttempts:  3,
            Backoff:      flowdsl.ExponentialBackoff,
            InitialDelay: 2 * time.Second,
        },
        Handler: func(ctx context.Context, packet map[string]any) error {
            out, err := nodes.CreateUserAccount(ctx, flowdsl.NodeInput{Packet: packet})
            if err != nil { return err }
            return bus.Publish(ctx, "flow.user_registration.send_verification", out.Packet("out"))
        },
    })

    // Edge: CreateUserAccount → SendVerificationEmail [durable, retry x3]
    bus.Subscribe(flowdsl.DurableConsumer{
        Topic:   "flow.user_registration.send_verification",
        GroupID: "flow.user_registration",
        RetryPolicy: flowdsl.RetryPolicy{MaxAttempts: 3, Backoff: flowdsl.ExponentialBackoff, InitialDelay: 2 * time.Second},
        Handler: func(ctx context.Context, packet map[string]any) error {
            out, err := nodes.SendVerificationEmail(ctx, flowdsl.NodeInput{Packet: packet})
            if err != nil { return err }
            return bus.Publish(ctx, "flow.user_registration.publish_registered", out.Packet("out"))
        },
    })

    // Edge: SendVerificationEmail → PublishUserRegistered [stream, topic: users.registered]
    bus.Subscribe(flowdsl.DurableConsumer{
        Topic:   "flow.user_registration.publish_registered",
        GroupID: "flow.user_registration",
        Handler: func(ctx context.Context, packet map[string]any) error {
            return nodes.PublishUserRegistered(ctx, flowdsl.NodeInput{Packet: packet}, "users.registered")
        },
    })
}

What changes when you edit the diagram

Change in StudioGenerated diffGo code change needed
Change direct → durable on an edgebus.Subscribe(DurableConsumer{...}) addedNone
Add retry policyRetryPolicy{...} in consumerNone
Add idempotency keyIdempotencyKey: func(p) ... addedNone
Change stream topic nametopic: "new-name" in consumerNone
Add new nodeNew field in XxxFlowNodes structImplement the handler (build fails until you do)
Remove a nodeField removed from structRemove the handler assignment (build warns)
Rewire edge from A→B to A→CConsumer topic changesNone (if C handler already exists)

Module Node Contract

Each module that contributes nodes to flows implements FlowNodeProvider:

go
// go-framework/modules/module.go (new interface)

// FlowNodeProvider is implemented by modules that register FlowDSL node handlers.
// The generated XxxFlowNodes struct is populated from these registrations.
type FlowNodeProvider interface {
    // FlowNodes returns all operationId → handler mappings this module provides.
    // Called at startup by flows.RegisterAll().
    FlowNodes() map[string]flowdsl.AnyHandler
}

Module implementation pattern

text
backend/modules/community/
  handler.go      ← HTTP handlers (unchanged)
  service.go      ← business logic (unchanged)
  flows.go        ← NEW: FlowDSL node handler wrappers
                         same Service calls, different transport adapter
go
// backend/modules/community/flows.go

// FlowNodes implements modules.FlowNodeProvider.
func (m *Module) FlowNodes() map[string]flowdsl.AnyHandler {
    return map[string]flowdsl.AnyHandler{
        "receive_registration":          flowdsl.Source(m.flowReceiveRegistration),
        "validate_registration_input":   flowdsl.Router(m.flowValidateRegistration),
        "route_registration_by_type":    flowdsl.Router(m.flowRouteByAccountType),
        "create_user_account":           flowdsl.Node(m.flowCreateUserAccount),
        "send_verification_email":       flowdsl.Node(m.flowSendVerificationEmail),
        "publish_user_registered":       flowdsl.Publish(m.flowPublishUserRegistered),
        "reject_invalid_registration":   flowdsl.Terminal(m.flowRejectInvalidRegistration),
    }
}

// flowCreateUserAccount wraps m.Service.Register — same logic as the HTTP handler,
// different input adapter (NodeInput instead of http.Request).
func (m *Module) flowCreateUserAccount(ctx context.Context, in flowdsl.NodeInput) (flowdsl.NodeOutput, error) {
    var req CommunityRegisterInput
    if err := in.Decode(&req); err != nil {
        return flowdsl.NodeOutput{}, err
    }
    member, user, err := m.Service.Register(ctx, &req)
    if err != nil {
        return flowdsl.NodeOutput{}, err
    }
    return flowdsl.Out("out", map[string]any{
        "id":           member.GetID().Hex(),
        "email":        user.Email,
        "account_type": string(member.AccountType),
    }), nil
}

The pattern: HTTP handlers and FlowDSL node handlers share the same Service layer. Only the transport adapter differs — httputil.ReadJSON(r, &input) becomes in.Decode(&input).


CI Pipeline

yaml
# .github/workflows/ci.yml

- name: Validate FlowDSL diagrams
  run: go run ./cmd/flows/ validate
  # Fails if any .flowdsl.yaml has invalid syntax or broken references

- name: Regenerate flow bridge code
  run: go run ./cmd/generate/ flows
  # Reads backend/flows/*.flowdsl.yaml → writes backend/flows/gen/*_flow_gen.go

- name: Check for uncommitted codegen drift
  run: git diff --exit-code backend/flows/gen/
  # Fails if .flowdsl.yaml was changed without regenerating
  # This enforces: diagram edit → commit → regenerate → commit gen → PR

- name: Build
  run: go build ./...
  # Fails if any operationId in a generated XxxFlowNodes struct is unimplemented

- name: Test
  run: go test ./...

The drift check is the key gate: if a developer edits a .flowdsl.yaml in Studio but forgets to regenerate, CI catches it before merge. The generated files are committed to git, making the diagram's impact on wiring fully visible in PR diffs.


Generator CLI: backend/cmd/flows/

text
go run ./cmd/flows/ validate           Validates all .flowdsl.yaml files (spec correctness)
go run ./cmd/flows/ check              Checks all operationIds have registered handlers
go run ./cmd/flows/ generate           Regenerates backend/flows/gen/*_flow_gen.go
go run ./cmd/flows/ scaffold <flow>    Generates handler stubs for unimplemented nodes in a flow
go run ./cmd/flows/ diff               Shows what a diagram edit changed in human-readable form
go run ./cmd/flows/ push               Uploads flows to Redelay cloud API

Startup Wiring

The generated RegisterAll function wires all flows using the node registry:

go
// backend/flows/gen/register_all_gen.go  ← generated
package flows

func RegisterAll(bus flowdsl.EventBus, reg *modules.Registry) {
    // Collect all node handlers from FlowNodeProvider modules
    nodes := flowdsl.BuildNodeRegistry(reg)

    RegisterUserRegistrationFlow(bus, UserRegistrationFlowNodes{
        ReceiveRegistration:        nodes.Require("receive_registration"),
        ValidateRegistrationInput:  nodes.RequireRouter("validate_registration_input"),
        RouteRegistrationByType:    nodes.RequireRouter("route_registration_by_type"),
        CreateUserAccount:          nodes.Require("create_user_account"),
        SendVerificationEmail:      nodes.Require("send_verification_email"),
        PublishUserRegistered:      nodes.RequirePublish("publish_user_registered"),
        RejectInvalidRegistration:  nodes.RequireTerminal("reject_invalid_registration"),
    })

    RegisterModuleSubmissionFlow(bus, ModuleSubmissionFlowNodes{ ... })
    RegisterEmailNotificationFlow(bus, EmailNotificationFlowNodes{ ... })
    // ...
}

In cmd/api/main.go:

go
func main() {
    inst, srv, err := server.Default()
    if err != nil { log.Fatal(err) }

    // Wire all FlowDSL flows after module startup
    if err := inst.AfterStart(func(ctx context.Context) error {
        return flows.RegisterAll(inst.EventBus, inst.Modules)
    }); err != nil {
        log.Fatal(err)
    }

    if err := app.Run(inst, srv); err != nil { log.Fatal(err) }
}

Cloud Deployment Model

Today (Git-deployed)

text
edit YAML in Studio → export to backend/flows/ → commit → CI runs generator
→ go build → docker build → docker push → container restart picks up new wiring

Later (Cloud-managed flows)

text
Studio → PUT /api/flows/{id} → Redelay cloud stores YAML
→ cloud runs generator → cloud rebuilds and redeploys container automatically
→ OR: container watches /flows/ endpoint and triggers self-restart on change

The cloud layer owns:

  • Kafka topic provisioning for stream edges
  • MongoDB collection provisioning for durable edges
  • Redis stream provisioning for ephemeral edges
  • Auto-scaling workers proportional to queue depth per edge

What to Build (Ordered)

#ComponentLocationUnblocks
1FlowNodeProvider interfacego-framework/modules/module.goModule node registration
2flowdsl.NodeInput/Output/AnyHandler typesgo-framework/flowdsl/node.goHandler function signatures
3flowdsl.NodeRegistrygo-framework/flowdsl/registry.goNode discovery at startup
4Flow YAML parser (FlowDSL.com format)go-framework/flowdsl/spec/Generator reads flows
5cmd/flows/ validatebackend/cmd/flows/CI gate on diagram validity
6cmd/generate/ flowsbackend/cmd/generate/Codegen from YAML
7Generated XxxFlowNodes structsbackend/flows/gen/ (generated)Compile-time enforcement
8flows.go node wrappers per modulebackend/modules/*/flows.goHandlers wired to flows
9flows.RegisterAll()backend/flows/gen/ (generated)Startup wiring
10cmd/flows/ scaffoldbackend/cmd/flows/Auto-stub for new nodes
11Direct edge executiongo-framework/flowdsl/Simplest flow paths
12Durable edge execution (go-events)go-framework/flowdsl/Reliable business paths
13Stream/Ephemeral edge executiongo-framework/flowdsl/Full delivery mode support
14/api/flows/ module + Studio integrationgo-framework/modules/flows/Cloud diagram management