Architecture outline: XML processing pipeline

The large XML file arrives through a defined intake channel, for example an SFTP folder, object storage (S3/Blob) with an event trigger, or a message queue. The ingestion service accepts the file, assigns a unique processing id (correlation id) and creates an entry in a status table (“received”). From this point on the file can be traced through the system.

One thing to watch out for with FTP/SFTP as the source, when files are not deleted right after processing: to avoid downloading and processing every file again on each poll, a file should either be moved into a folder such as “processed/” or renamed on the server (where write permissions allow), or — if the files have to stay where they are, untouched — the directory metadata (file name, size, modification date) should be compared against the existing status table before a file is downloaded. The move or rename is best done at the end of stage 1, as soon as parsing and persistence into the database have completed successfully — not only after the processing logic and XML generation, because those are separate, decoupled process chains that only read from the database anyway and no longer need the source file from that point on. The directory listing itself is cheap; only downloading and processing again is expensive. A watermark (the modification date of the last successfully processed file) helps to keep the comparison set small, a claim mechanism helps when several ingestion workers run in parallel, and idempotent persistence acts as a backstop against accidental double processing.

Because the files are large, a streaming parser is used (SAX/StAX in Java or iterparse in Python, for instance) instead of loading the file into memory as a whole. The parser splits the XML structure into meaningful business units — individual records, say — and passes them on one by one rather than waiting for the end of the file.

Reading itself stays sequential, since the parser walks through the token stream byte by byte to recognise element boundaries. But as soon as it has recognised a complete business unit (a full element with all its child elements, for example), the further processing of that unit — validation, transformation, writing to the database — can run in parallel. This is the classic producer–consumer pattern: the parser thread keeps “producing” finished units while several consumers work through them side by side.

The split units are persisted, ideally with batch inserts rather than individual ones, so that large volumes stay manageable. Business fields are held relationally; if parts of the original XML structure have to be preserved, an additional blob or document store for raw fragments can make sense.

A message queue (Kafka or RabbitMQ, for instance) sits between persistence and the processing logic. It decouples the steps in time, allows the following workers to scale horizontally and makes the pipeline more robust against load peaks — including a retry mechanism and a dead letter queue for units that keep failing.

One open question here: who creates the message and puts it into the queue? A direct dual write (first the database insert, then a separate queue publish within the same processing step) is risky, because the two are independent systems without a shared transaction — if either step fails, database and queue are inconsistent.

Recommended: the transactional outbox pattern. In the very same database transaction in which it writes the business data, the writing consumer also creates an entry in an outbox table — making it atomic, either both or neither. A separate, slim relay process continuously reads that outbox table (polling), publishes the entries to the queue and marks them as sent afterwards.

An alternative without a real message broker: a scheduler periodically queries the database for records with the status “raw, not yet processed” (pull instead of push). Technically not a queue in the classic sense, but operationally the simplest option and often good enough for moderate throughput.

A note: if Spring Batch is used for this, parallelisation can happen internally through a multi-threaded step or partitioning — and then the queue can genuinely be saved. The upside: considerably less infrastructure, no broker to operate, built-in restart and retry logic through the job repository. The downside: parallelisation only scales within one JVM/one machine — real horizontal scaling across several instances needs coordination of its own (“SELECT ... FOR UPDATE SKIP LOCKED”, for example), which a queue with consumer groups brings along out of the box. On top of that a scheduler works in cycles (batch latency) while a queue reacts immediately — which matters if near real-time processing or several independent consumers of the events are needed. For a single instance with moderate throughput and tolerable batch latency, the Spring Batch variant is a sensible simplification.

One or several worker services read the persisted data and carry out the actual business processing logic. Because the queue decouples them, several worker instances can run in parallel to raise throughput when needed.

A note: a message-driven bean (MDB) works well as a worker if a Java EE/Jakarta EE container (WildFly, Open Liberty or similar) is in use — the container invokes it automatically when a message arrives and can tie the processing transactionally to the database write. With Spring/Spring Boot the functional counterpart is a method annotated with `@JmsListener` or `@KafkaListener`; in both cases scaling happens through the number of concurrent consumer instances, not through additional processes.

The computed results are stored in result tables of their own, linked to the original records through the correlation id. That keeps it traceable which result belongs to which incoming file and record.

A service reads the result data from the database and builds the new XML structure from it, for example through templating/XSLT or object-to-XML mapping such as JAXB. Here too a streaming approach is advisable if the result files can grow large again.

“Templating” here means that the output structure is defined up front as a template with placeholders, which a template engine (FreeMarker or Velocity, for instance) fills with the result data at runtime — as opposed to building the XML programmatically in code. An example of such a template: For comparison: XSLT transforms an existing XML structure into another one through declarative rules (useful when the result data is already available as XML), while JAXB (object-to-XML mapping) needs no template at all — annotated Java classes are serialised straight into XML.

The generated XML result file is placed at a defined outgoing point (folder, object storage, SFTP outbox) and the processing status is set to “completed”. Downstream systems are notified of completion where required.

Instead of one service class calling several processing classes explicitly one after another, the pipeline or chain-of-responsibility pattern is the better fit: a shared interface for a processing step, with the concrete steps (validation, enrichment, computation, result mapping, for example) lined up in a list that a pipeline walks through in order. The benefit: every step can be tested on its own, new steps can be added without touching existing classes (the open/closed principle), and the order is visible in one place instead of being scattered across method calls.

If the actual computation differs depending on the kind of business unit (different document or record types, each with its own computation rules, say), add the strategy pattern on top instead of cascades of if/else or switch. One interface, BerechnungsStrategy for instance, with several implementations per type, selected through a registry or factory — in Spring, for example, through an injected List<BerechnungsStrategy> from which the matching implementation is determined via an unterstuetzt(typ) method, entirely without central branching logic that grows with every new type.

A note: if the computation is very complex and rich in business rules, a richer domain model (in the DDD spirit) is worth considering as well — putting the rules directly onto the business objects rather than packing everything procedurally into processing classes, so that the objects do not degenerate into pure data containers (“anemic domain model”). For a broad outline, though, pipeline + strategy is entirely sufficient.

As an alternative or a complement to the purely in-memory pipeline: every record gets an explicit, persisted status (an enum column) with a fixed set of permitted transitions rather than a free-text field — RECEIVED → PERSISTED → VALIDATED or REJECTED → COMPUTED → XML_GENERATED → COMPLETED, for example. Validation is then the first transition: to VALIDATED in the positive case, to REJECTED in the negative one — and only in the VALIDATED state does the record become visible to the next stage. Technically a status column plus a history/event table (old status, new status, timestamp, error message where applicable) is usually enough; with more complex state logic (repeats, parallel partial states, manual intervention) a real state machine library such as Spring State Machine pays off. The benefit: a GUI almost falls out of this by itself — an overview of how many records currently sit in which state, and per record the complete transition history to trace what happened. This lines up with the status and error tracking already planned, but formalises it into a genuine state model. Worth keeping in mind: the more states there are, the more important a state diagram thought through in advance with clearly permitted transitions, otherwise it drifts over time into an unmanageable sprawl of statuses.

More states do not automatically mean more infrastructure components (MDBs/listeners). State is a cheap data model concept, whereas an MDB/queue is a more expensive piece of infrastructure that has to be deployed, monitored and scaled — the two need not grow one to one. The MDB belongs at the deliberately chosen decoupling boundaries (the existing stage boundaries 1 → 2 → 3), not at every single intermediate state; within one consumer invocation a record can happily travel through several intermediate states without another message hop. To select the right handler per state, a registry suggests itself — much like the strategy above: an interface ZustandsHandler with one implementation per state/transition, selected through a map from state to handler. A new state then means one new handler class plus a registry entry, no new queue and no new listener.

The exception: if a particular transition should deliberately scale independently or be monitored in isolation (a compute-heavy calculation kept separate from validation, say), a queue/MDB of its own does make sense — but that should then be a conscious scaling or isolation decision, not an automatic consequence of additional states.

Technical errors (database unreachable, NullPointerException, timeout) are unexpected — an exception with a transaction rollback and retry/dead letter queue is the right answer there. Business errors (inconsistent data in the XML, a missing mandatory field, a reference to an unknown master record) are not programming errors, though, but an expected, valid outcome of validation, and should not be modelled as an exception with a rollback. Otherwise two things go wrong: the rollback also throws away the information that the record was rejected and why, and with retry/redelivery through the queue the classic “poison message” problem looms — a record that will never become valid but keeps being delivered again endlessly.

The cleaner approach: validation returns a result object rather than an exception (the notification pattern) — a list of validation errors with field, rule and message text, for instance. That fits the pipeline pattern above directly: the validation step is the first step in the pipeline and returns either “OK” or “business error plus error list”. On a business error the orchestrating service skips the following steps (processing logic, computation) but commits the record perfectly normally — just with a status of its own such as “rejected”/“business error” instead of “processed successfully”, together with the structured error details. Important for the status model: a technical error is retryable, a business error is a terminal but valid state that should not be escalated as if it were a malfunction.

Informing the submitter: the mechanism that exists for the result XML anyway (stage 3, XML generator) lends itself to this. An additional or alternative feedback/rejection XML type lists per record which rule was violated and why — for that, the XML generator reads the records with the status “rejected” instead of “computed successfully” and turns them into a rejection receipt for the submitter. Structurally the same process as the normal result file, only with a different template and a different data source. Where required, an email notification or a report/dashboard for the business department on top, for cases such as a spike in rejections.