Amish Kushwaha - October 20, 2026

xpro is now open source

We have been running a small XML ingestion service in production for a while. It reads files, pulls out the fields you care about, and writes them into a database quickly and safely. Today we are releasing a cleaned-up, generalized version of it as open source under the Apache-2.0 license.

It is called xpro-oss, and you can find it here: github.com/bluefunda/xpro-oss

Where it came from

If you work with SAP data, you know the pattern. Documents show up as XML files, maybe billing exports, maybe IDoc dumps, maybe the output of an ABAP transformation. They sit on an SFTP server or in a folder, and someone has to get them into a database so people can actually use them.

The first version of that job is easy to write. The second and third versions are where it gets annoying. Volumes grow. One broken file takes down a whole batch. The files turn out to live behind a network you cannot reach directly. And every new XML shape means another round of parser code.

We built xpro to stop writing that job over and over. The version we ran internally was tied to one schema, so for the open source release we rebuilt it from scratch with nothing hardcoded. No schema, no table name, no SAP-specific idea baked into the Go code. Everything lives in one config.yaml.

What it does

xpro reads XML files from a source, maps XPath expressions onto database columns, and inserts the results in batches. That is the whole idea. Here is what a mapping looks like:

mapping:
  - column: account_number
    xpath: "/Root/Account/Number"
  - column: amount
    xpath: "/Root/Billing/Amount"
  - column: raw_xml
    xpath: ""

An empty XPath keeps the raw file, which is handy when you want the original document next to the extracted fields. If your XML changes shape, you edit the mapping. You do not rebuild anything.

The features we care about most

It does not let one bad file ruin the batch. Each batch goes in as one transaction, but every row is isolated. On PostgreSQL and Oracle that is done with a savepoint per row. SAP HANA has no savepoint statement, but it rolls back a failed statement on its own and keeps the transaction alive, so xpro simply records the failure and keeps going. Either way, the good rows commit and the bad ones are reported by file name.

Nothing disappears quietly. Every file is accounted for. Inserted plus failed always equals the number of files read. If a file cannot be read or parsed, you will see it in the results.

Retries are careful. A batch is retried once, and only when the whole transaction failed because of something transient, like a dropped connection or a timeout. It never retries after a partial failure or an uncertain commit, because that is how you end up with duplicate rows.

Three databases, no client libraries. PostgreSQL, Oracle and SAP HANA are all supported, and all three drivers are pure Go. There is no CGO, no Oracle Instant Client and no HANA client to install. You get one static binary and a small container image.

Local folders or SFTP. Point it at a directory, or at an SFTP server with optional host key verification.

SAP BTP support when you want it. If your SFTP server or database is reachable through SAP BTP, xpro can resolve it through the Destination service and tunnel through the Connectivity service. This only switches on when the bindings are present. Outside BTP, you use a plain host and port and nothing else changes. It runs fine on AWS, GCP, bare metal or a systemd unit.

Simple to operate. It is a small HTTP service with health and readiness endpoints, a /process endpoint to start a run, and a /status endpoint to see how the last run went. Only one run happens at a time, so you cannot accidentally insert every file twice. The config is strict, so a typo in a key is an error instead of a silent change in behavior.

How fast is it?

Fast enough that we stopped worrying about it. We benchmarked the same pipeline the service uses, with about 20,000 SAP ABAP-XML billing documents of roughly 19 KB each, on a single 16 vCPU test machine with the database on the same host. These are the best results with zero failed rows:

Throughput scaled about 7.5 times going from 1 to 16 workers, then levelled off. A good starting point is 16 workers with a batch size of 100. Extrapolating from those numbers, a run of 300,000 small files would take roughly 35 to 55 seconds, depending on the database.

A few honest caveats. This was one machine, with local files and a warm cache. HANA express and Oracle Free are capped editions, and the PostgreSQL run used default settings. So please do not read the table as a ranking of databases. With remote storage or a remote database, expect it to be slower, and measure on your own setup. The full method and raw results are in the repository, and you can reproduce them with go run ./cmd/bench.

Where it is useful

• Landing SAP XML exports, such as billing or ABAP-XML output, into HANA, PostgreSQL or Oracle for reporting
• Loading files from an SFTP drop on a schedule, including ones behind a BTP tunnel
• One-off migrations where you need to get a large pile of XML into a table without writing a custom loader
• A staging step in a bigger pipeline, where you keep the raw XML and a few extracted columns

Try it

There are Docker Compose examples for all three databases. With HANA, for instance:

git clone https://github.com/bluefunda/xpro-oss
cd xpro-oss/examples/hana
cp .env.example .env
docker compose up -d
curl -X POST "localhost:8080/process?wait=true"

That starts the database, applies the schema, and loads the sample XML. The running guide in the repo walks through configuration, tuning and troubleshooting.

Why open source

Honestly, because this kind of tool is not special to us. Plenty of teams write the same loader, and we would rather share a solid one than have everyone start from zero. It is also a small codebase on purpose, mostly standard library, so it is easy to read and easy to change.

If you want to help, there is plenty to do. New sources like S3 or Azure Blob, new database targets, support for files that contain several documents, and benchmarks from your own hardware would all be welcome. Open an issue or send a pull request, and let us know what you end up loading with it.

• Repository: github.com/bluefunda/xpro-oss
• Container image: ghcr.io/bluefunda/xpro-oss
• License: Apache-2.0