The Internet Is Full of Duplicate Requests—Here’s How Smart Developers Prevent Them | HackerNoon

When developing web applications, securing requests between the frontend and backend is one of the key tasks.

Ignoring this aspect can lead to serious consequences: data leaks, duplicated operations, incorrect financial transactions, and even service outages. In this article, we will examine the main methods of securing requests and take a detailed look at such an important mechanism as idempotency and how to implement an idempotency key in web applications.

Before diving into idempotency, let’s review the basic methods of securing requests:

1. JWT Tokens — Used to authenticate users and verify the authenticity of a request based on a signature.
2. HTTPS — Protects data from interception by encrypting traffic between the client and server.
3. CSRF Tokens — Protect against cross-site request forgery attacks.
4. Data Validation — Prevents attacks by validating incoming data for correctness.
5. Rate Limiting — Limits the number of requests from a single source per unit of time, protecting against DDoS attacks.
6. CORS Policies — Allow controlling which domains are permitted to send requests to the server.
7. Password Hashing — Ensures the security of stored passwords.
8. HttpOnly/Secure Cookies — Protect session data from theft through XSS attacks.
9. Data Sanitization — Protects the server from attacks involving malicious code execution.
10. Request Signing — Ensures data integrity during transmission.
11. Two-Factor Authentication (2FA) — Adds an additional level of security for critical operations.

These measures create a baseline level of security, but they do not protect against request duplication.

For that, idempotency is used.

Let’s take a closer look at what that means.

What is Idempotency?

Idempotency is the property of an operation where performing it multiple times produces the same result as performing it once.

Example:

• If a client sends two identical requests to withdraw funds, the funds should only be withdrawn once.

• If a client accidentally creates two identical orders, the server should create only one order.

To implement idempotency, the client must send a special header Idempotency-Key — a unique operation identifier — with the request. The server checks this key and decides whether to process the request or not.

To implement idempotency, the following principles must be followed:

1. The client must generate the idempotency key before sending the request.
2. The server stores the key value and the result of request processing.
3. If a request with the same key has already been processed, the server returns the stored result without reprocessing.

let’s go deeper!

Generating an Idempotency Key on the Client

The idempotency key should be generated on the client side because only the client knows which requests might be duplicates.

The key should meet the following requirements:

• Be unique for each request.
• Remain consistent when the same request is retried (e.g., after a network failure).
• Should not depend on random factors (e.g., time).

Methods for Generating a Key:

1. UUID (v4) This is the most common method. A UUID (Universally Unique Identifier) is a random 128-bit string.

const idempotencyKey = crypto.randomUUID();

2. Hash of Request Data If requests contain identical parameters, you can generate a key as a hash of the request data. This makes the key deterministic.

const data = JSON.stringify({ userId: 123, amount: 500 });
const idempotencyKey = crypto.createHash('sha256').update(data).digest('hex');.

3. Combination of Timestamp and Random DataA key can be created from a timestamp (to prevent collisions) and a random number.

const idempotencyKey = `${Date.now()}-${Math.random().toString(36).substring(2)}`;

Processing the Key on the Server

After the client sends a request with the Idempotency-Key header, the server performs the following actions:

1. Extracts the key from the header:

key := r.Header.Get("Idempotency-Key")
if key == "" {
    http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
    return
}

2. Checks whether such a key already exists in the storage:

3. If processing is successful:

• Stores the result and key in the cache or database.
• Sets TTL (time-to-live) to automatically delete the key after a defined period.

Example Implementation in Go

var idempotencyCache = sync.Map{}
var cacheTTL = 24 * time.Hour

func WithIdempotency(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        key := r.Header.Get("Idempotency-Key")
        if key == "" {
            http.Error(w, "Idempotency-Key required", http.StatusBadRequest)

            return
        }

        if cached, ok := idempotencyCache.Load(key); ok {
            w.WriteHeader(http.StatusOK)
            w.Write(cached.([]byte))

            return
        }
        
        // your logic

        if recorder.statusCode >= 200 && recorder.statusCode < 300 {
            idempotencyCache.Store(key, recorder.body)
            time.AfterFunc(cacheTTL, func() {
                idempotencyCache.Delete(key)
            })
        }
    })
}

Conclusion

Idempotency allows you to avoid request duplication and make the system reliable in the event of network failures and repeated calls. Implementing it via middleware in Go simplifies request processing and increases system resilience.

However, you need to account for possible mistakes during implementation:

1. TTL too short — The key may disappear before request processing is complete.
2. Non-unique keys — If the key is reused across different operations, the server may return the wrong cached result.
3. Caching errors — Do not cache responses with HTTP 500 or similar errors.

Idempotency is a simple yet powerful tool for preventing duplicate operations.

Start using it in all unsafe HTTP methods (POST, PUT, PATCH) — and you will make your API more stable and predictable.

Take care!