Blog

Integrating your software: APIs, synchronization and the end of double data entry

For organizations · By KTMA SOLUTIONS · Published · 9 min read

If someone in your organization re-types data from one piece of software into another every week, you already know the problem. The question is no longer "should we integrate our software?" but "how do we do it without creating a new weak point?" This article explains the mechanisms (APIs, webhooks, files), the underlying decisions (direction of synchronization, source of truth) and the requirements that separate an integration that lasts from one that breaks at the first incident.

The symptoms of double data entry

Double data entry is rarely recognized by name. It shows up instead in very concrete ways:

  • the same client record exists in two slightly different versions depending on which tool you open;
  • an "official" report is redone by hand because nobody trusts the automatic figures;
  • a re-keying task comes back every month-end, with a designated person who knows "the tricks";
  • a typing error discovered late has to be corrected in several places;
  • a new employee slows everything down, because the procedure lives in someone's head.

These symptoms share a common source: several systems hold the same information with no controlled link between them. The goal of an integration is therefore not "to make software talk to each other," but to decide who holds what and to make the transfer reliable.

Three ways to integrate your software

APIs

An API (application programming interface) lets one piece of software query or modify another according to published rules. On the Web, this is most often an HTTP API. The OpenAPI Initiative describes the OpenAPI Specification as "a vendor-neutral description format" for remote HTTP-based APIs: it lists the permitted interactions, the parameters, the return values and the data formats. According to the initiative, such a description makes it easier to generate documentation consistent with the API, to generate starter code and to create mock servers so you can test early, even before the provider's code is written.

For an organization, the lesson is practical: before choosing an approach, check whether the software involved publishes an up-to-date API description. It is a good indicator of how easy (and how costly) integration will be.

Webhooks

A webhook reverses the logic: instead of regularly polling the other system, you give it an address, and it notifies you when an event occurs. The documentation from Stripe, a payment provider that makes extensive use of this mechanism, illustrates the precautions to plan for. Its guide asks you, among other things, to verify the signature of each request, to respond quickly with a success code (2xx) before any complex processing, and to process events asynchronously, through a queue, to withstand volume spikes. It also states that delivery order is not guaranteed and that the same event may arrive more than once.

These principles apply well beyond that provider: every webhook receiver must be designed for duplicates, out-of-order delivery and authenticity checks.

File exchanges

Some systems, often older ones, offer neither an API nor webhooks, but can export or import files (CSV, XML, etc.). A file-based integration can be perfectly reasonable for modest volumes or periodic exchanges, provided it addresses the same topics as an API: a documented format, validation on receipt, duplicate checks, a processing log. The main risk is dependence on one person who "drops the file in the right place."

One-way or two-way synchronization

This is the design decision that has the greatest impact on the overall structure.

One-way: one system sends, the other receives. It is simpler to understand, test and troubleshoot. It suits cases where a piece of information clearly originates in a single place, for example an order entered in one tool and then picked up by billing.

Two-way: both systems can modify the same information and must synchronize in both directions. This is sometimes necessary, but it raises difficult questions: what happens if the same record is modified on both sides before the next synchronization? Which change wins? On what basis is that decided?

A prudent rule is to start with one-way and move to two-way only when a clear business need requires it, field by field rather than for the whole object.

Designating a source of truth

For each shared piece of data (client, product, employee, invoice) and, where needed, for each field, designate a source of truth: the system authorized to create or correct that data. The other systems receive a copy of it, according to documented synchronization rules. In practice:

  1. list the shared data;
  2. assign an owning system to each item;
  3. document the matching rules (for example, which identifier links a client in tool A to its counterpart in tool B);
  4. prohibit changes to fields managed elsewhere in the secondary systems, or make such changes visible as an exception.

Without this decision, no technology will keep new data discrepancies from appearing.

Handling errors and recovery

An integration will eventually fail: network outage, unavailable service, invalid data, format change. The quality criterion is not the absence of errors but the behaviour when one occurs.

Making operations repeatable

The MDN glossary defines an idempotent HTTP method as one whose intended effect on the server is the same whether the request is sent once or several times. The GET, HEAD, OPTIONS, PUT and DELETE methods are idempotent; POST and PATCH are not guaranteed to be idempotent. MDN draws the conclusion: a client can safely retry an idempotent request when it does not know whether it reached the server. Conversely, repeating a POST may create a duplicate (MDN's example is adding a row each time the request is sent).

For an integration, this means designing exchanges so that a replay duplicates nothing: unique identifiers, updates by replacement rather than blind creation, and, on the webhook side, keeping the identifiers of events already processed, as Stripe recommends.

Respecting the provider's limits

When a service judges that you are sending too many requests, it may respond with 429 ("Too Many Requests"), a mechanism called rate limiting. It may add a Retry-After header indicating the waiting time, in seconds or as a date. MDN notes that support for this header varies across clients and servers. When it is present and valid, respect the indicated delay; if it is absent or unusable, plan a default progressive back-off delay.

Planning a queue for failed cases

A good integration does not simply retry indefinitely. It limits the number of attempts, sets aside the cases that keep failing, and alerts a designated person with enough context to fix them. It also lets you replay a batch once the underlying problem has been corrected.

Securing access

Connecting two pieces of software opens a door between them. OWASP, a recognized application-security organization, maintains a list of the ten main risks specific to APIs (2023 edition). Several bear directly on integrations:

  • Object-level authorization: an endpoint that receives an identifier may allow access to data that should not be accessible.
  • Broken authentication: weak handling of logins or tokens allows an identity to be impersonated.
  • Property-level and function-level authorization: missing checks on certain fields or functions expose or modify data without permission.
  • Unrestricted resource consumption: without usage limits, an API can be overloaded or can run up high costs when it relies on paid services.
  • Unsafe consumption of APIs: too much trust is placed in the data from a third-party API, which attackers may target to reach your system.

This last point is often overlooked: data you receive from a partner must be validated like any other input.

On the practical side, at a minimum, keep in mind:

  • a dedicated service account for the integration, with strictly the necessary rights (the principle of least privilege);
  • secrets (keys, tokens) stored outside the code and renewable;
  • encrypted connections (HTTPS/TLS) between systems: MDN warns that Basic authentication, which encodes the username and password in a reversible way, would be "completely insecure" without a secure connection (HTTPS/TLS);
  • verification of the authenticity of incoming calls, for example webhook signatures.

Logging to understand, without exposing

When a client asks "why did this order never arrive?", only a log can answer. The OWASP logging guide recommends recording security events (authentication or authorization failures, validation failures), application errors, configuration changes and access to, or export of, sensitive data.

The same guide asks that passwords, access tokens, keys, connection strings and sensitive personal information not be written in clear text in logs: they must be removed, masked or hashed. It also recommends controlling who can read the logs, detecting any tampering and using structured formats rather than free text. In plain terms: a log that is useful for operations must not become a second data leak.

Testing before, during and after go-live

An integration is tested on several levels:

  1. Nominal cases: valid data flows correctly from one system to the other.
  2. Edge cases: empty fields, accented characters, very large volumes, duplicates.
  3. Simulated failures: service unavailable, slow response, 429, replay of the same event, events out of order.
  4. Reconciliation: periodic comparison of the two systems to detect any silent discrepancy.
  5. Test environment: if the provider offers one, use it before involving real data.

The OpenAPI description, when it exists, helps build these tests: the OpenAPI Initiative points out that it makes it possible to simulate the API to test early.

Planning for maintenance

An integration is never "finished." Providers evolve their APIs, keys expire, volumes change, and so do business processes. Plan from the start for:

  • a person responsible for the integration, with a backup;
  • up-to-date documentation of the flows, field mappings and recovery procedures;
  • monitoring of providers' change notices;
  • an inventory of the integrations in place (OWASP, incidentally, ranks poor API inventory management among its ten risks);
  • alerts on repeated failures.

Frequently asked questions

Do you always have to go through an API? No. An API is generally preferable when one exists, but a well-managed file exchange can be enough for simple needs or older software.

Where should you start? With an inventory of current duplicate data entry tasks: who re-enters what, how often, and with which observed errors. This exercise shows where integration brings the most value and in what order to act.

How do you avoid duplicates in an automated exchange? With unique identifiers, operations designed to be replayed without side effects, and, on the webhook side, by keeping the identifiers of events already processed.

Conclusion

Integrating your software is above all an exercise in clarity: deciding who holds each piece of data, in which direction it flows, how you react when an exchange fails, who has access to it and how you keep a record of it. The technology (APIs, webhooks, files) comes afterward. If you have already identified where data is entered twice, you have done the hardest part: describing the problem.

KTMA solutions that can help

Have a specific situation in mind? Talk about my project.

Sources

Back to the blog · Blog RSS feed