Limiting Concurrent Uploads with a Promise Pool #

Start N async “workers” that each pull the next file from a shared iterator and upload it, looping until the iterator is exhausted; await Promise.all(workers) resolves when every file is handled. Wrap each upload in try/catch so a failure records a result for that file instead of rejecting the pool, pass one AbortSignal to every upload so cancelling the batch stops everything in flight, and check signal.aborted before taking the next file. That gives you at most N uploads at a time, next-file-starts-immediately scheduling, and per-file outcomes in the original order — in about thirty lines, with no library.

Promise.all(files.map(upload)) starts every upload at once; the browser then queues most of them behind its connection limit while their timeouts tick, and a single rejection makes the whole batch look failed. Chunking the list into groups of N (for loop over slices with Promise.all) bounds concurrency but wastes capacity: each group waits for its slowest file. A pool keeps exactly N busy at all times. This page belongs to upload queue concurrency control in frontend UX, chunking and progress tracking; the full queue with pause and priorities is in prioritizing and pausing items in an upload queue.

When to use this approach #

• A user selects or drops many files and you upload them in one go.
• You upload the chunks of one large file in parallel and need a limit per file.
• You want bounded concurrency without adopting a queue library.

Prerequisites #

1. An upload function (file, signal) => Promise<T> — fetch with a presigned URL, or your chunked uploader.
2. Browsers or Node 18+ (the code uses AbortController, async iterators and Promise.all).
3. A decision on N; 3 for mixed media, up to 6 for many small files.

Batches versus a pool #

Implementation #

export type Outcome<T> =
  | { status: "fulfilled"; value: T; index: number }
  | { status: "rejected"; reason: unknown; index: number }
  | { status: "skipped"; index: number };

export async function pool<I, T>(
  items: Iterable<I>,
  limit: number,
  worker: (item: I, index: number, signal: AbortSignal) => Promise<T>,
  opts: { signal?: AbortSignal; onSettled?: (o: Outcome<T>) => void } = {},
): Promise<Outcome<T>[]> {
  const signal = opts.signal ?? new AbortController().signal;
  const results: Outcome<T>[] = [];
  const iterator = Array.from(items).entries()[Symbol.iterator]();

  async function run() {
    for (let next = iterator.next(); !next.done; next = iterator.next()) {
      const [index, item] = next.value;
      let outcome: Outcome<T>;
      if (signal.aborted) {
        outcome = { status: "skipped", index };
      } else {
        try { outcome = { status: "fulfilled", value: await worker(item, index, signal), index }; }
        catch (reason) { outcome = { status: "rejected", reason, index }; }
      }
      results[index] = outcome;
      opts.onSettled?.(outcome);
    }
  }

  const n = Math.max(1, Math.min(limit, Number.MAX_SAFE_INTEGER));
  await Promise.all(Array.from({ length: n }, run));
  return results;
}

Using it for a batch of files with presigned URLs:

const controller = new AbortController();
cancelButton.onclick = () => controller.abort();

const outcomes = await pool(files, 3, async (file, i, signal) => {
  const { url, headers } = await getUploadUrl(file, signal);
  const res = await fetch(url, { method: "PUT", body: file, headers, signal });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  return { key: url.split("?")[0] };
}, {
  signal: controller.signal,
  onSettled: (o) => ui.markRow(o.index, o.status),
});

const failed = outcomes.filter((o) => o.status === "rejected");
ui.summary(`${outcomes.length - failed.length} of ${files.length} uploaded`);

Line-by-line on the decisions that matter #

• Shared iterator. All workers call iterator.next() on the same object. JavaScript runs one worker at a time between awaits, so two workers can never take the same item; no locks are needed.
• Workers, not a counter. Starting limit loops that each process items until none remain is simpler and less error-prone than tracking an “active” count and chaining .then calls.
• Catch inside the loop. A rejected upload becomes an outcome for that index and the worker continues. Without this, one failure would reject Promise.all while the other workers kept uploading in the background with nobody listening.
• Results by index. Outcomes land in the same order as the input regardless of completion order, which makes mapping back to UI rows and file lists trivial.
• Skipped after abort. When the batch is cancelled, in-flight uploads reject with AbortError (because they received the signal) and remaining items are marked skipped without starting. The pool still resolves, so cleanup code runs.
• onSettled callback. The UI updates each row as it finishes instead of waiting for the whole batch.

Using the same pool for chunks #

Inside the worker for a large file, split it into parts and call pool(parts, 4, uploadPart, { signal }) with the signal the outer pool passed in. The inner pool’s outcomes tell you which parts failed; retry those (with backoff) before declaring the file failed. For S3 multipart, collect { PartNumber, ETag } from the fulfilled outcomes and complete the upload — S3 multipart upload orchestration covers the server side.

Retrying inside the pool #

Transient failures should be retried before they become outcomes. Wrap the worker in a retry helper rather than building retries into the pool, so the pool stays a scheduling primitive:

export async function withRetry<T>(fn: () => Promise<T>, signal: AbortSignal, attempts = 4): Promise<T> {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (e: any) {
      const permanent = e?.name === "AbortError" || /HTTP 4(0[0-4]|13)/.test(String(e?.message));
      if (permanent || i + 1 >= attempts) throw e;
      const delay = Math.min(30_000, 500 * 2 ** i) * (0.5 + Math.random());   // jittered exponential backoff
      await new Promise((r, j) => {
        const t = setTimeout(r, delay);
        signal.addEventListener("abort", () => { clearTimeout(t); j(signal.reason); }, { once: true });
      });
    }
  }
}

A retrying worker holds its slot while it waits. That is usually right — it keeps total concurrency honest — but with long backoffs and many failures, slots sit idle. If that matters, return the item to the end of the iterator instead of sleeping; the queue version in prioritizing and pausing items in an upload queue does exactly that with a failed → pending transition. Backoff strategy is covered in implementing exponential backoff for failed chunks.

Reporting progress from a pool #

A pool knows three things the interface needs: how many items are done, how many are running, and which failed. Keep counters next to the pool rather than scanning results on every render — increment running when a worker takes an item, move it to done or failed in onSettled — and derive the overall percentage from bytes, not item counts, so one large video does not make the bar sit at 99 % for minutes. Per-item byte progress comes from the upload function (XHR upload.onprogress, or acknowledged chunk sizes for chunked uploads); sum it across items for the total, as covered in aggregating progress across multiple files.

Emit counter changes at most once per animation frame. With several uploads reporting progress many times a second, updating the DOM on every event is the most common cause of a sluggish upload page; batching is covered in rendering smooth progress bars without jank.

Configuration gotchas #

Pool finishes but some uploads are still running. A worker threw outside the try (for example in getUploadUrl before the upload started, if you moved it out of the worker). Keep everything per-item inside the worker function.

Cancel does not stop uploads. The signal was not passed to fetch. Every network call inside the worker — URL request and upload — must receive it.

Total in flight is higher than expected. Nested pools multiply: 3 files × 4 parts = 12. Choose limits with the product in mind, and remember URL requests to your API count too.

Memory spikes with thousands of files. Array.from(items) holds references only, which is cheap. Spikes come from reading file contents eagerly (for hashing or previews) before upload; do that inside the worker.

Verification #

// Unit test: never more than `limit` concurrent, all items processed, order preserved.
let active = 0, peak = 0;
const out = await pool([...Array(50).keys()], 3, async (n) => {
  active++; peak = Math.max(peak, active);
  await new Promise((r) => setTimeout(r, Math.random() * 20));
  active--;
  if (n % 10 === 0) throw new Error("boom");
  return n * 2;
});
console.assert(peak === 3, "peak concurrency");
console.assert(out.length === 50 && out.filter((o) => o.status === "rejected").length === 5, "outcomes");
console.assert(out[7].status === "fulfilled" && out[7].value === 14, "order");

In the browser, drop 100 files and watch the network panel: exactly three uploads run at a time, and a new one starts the instant one finishes.

Frequently Asked Questions