Queues

Async job processing with named topics, retries, and dead-letter support via the iii-queue worker.

The iii-queue worker decouples producers from consumers: a function publishes a message to a named topic and returns right away, and any function subscribed to that topic processes the message in the background, with retries and a dead-letter queue (DLQ) for messages that keep failing.

iii worker add iii-queue

This page is a quick tour. For the full functionality of the queue worker see the queue worker docs.

Named Queues

When you need to control function execution for time consuming operations, or guarantee a certain number of retries then you can use TriggerAction.Enqueue to place that operation into a queue.

Creating a Queue

Queues are defined in the iii-queue worker’s config under queue_configs:

- name: iii-queue
  config:
    queue_configs:
      email-jobs:
        max_retries: 3
        concurrency: 10
        type: standard

Each queue_configs entry accepts:

Field	Default	Description
max_retries	3	Delivery attempts before the message moves to the DLQ.
concurrency	10	Jobs processed in parallel. Must be at least 1; the schema rejects 0, so a named queue cannot be paused this way. fifo queues force this to 1.
type	standard	standard (concurrent) or fifo (ordered within a message group).
message_group_field	none	Required for fifo: payload field whose value determines the ordering group.
backoff_ms	1000	Base retry delay in milliseconds; exponential.
poll_interval_ms	100	Worker poll interval in milliseconds.

Enqueue functions

Enqueued functions are registered the same as any other call to worker.trigger with the one difference being providing an action called TriggerAction.Enqueue to the trigger:

Node / TypeScript

import { TriggerAction, type EnqueueResult } from "iii-sdk";

const { messageReceiptId } = await worker.trigger<unknown, EnqueueResult>({
  function_id: "email::send",
  payload: { to: "[e-mail]", subject: "hi" },
  action: TriggerAction.Enqueue({ queue: "email-jobs" }), // "queue" specifies the name of the queue
});
// messageReceiptId identifies the enqueued job

Python

from iii import TriggerAction

receipt = worker.trigger({
    "function_id": "email::send",
    "payload": {"to": "[e-mail]", "subject": "hi"},
    "action": TriggerAction.Enqueue(queue="email-jobs"), # "queue" specifies the name of the queue
})
# receipt["messageReceiptId"] identifies the enqueued job

Rust

use iii_sdk::TriggerAction;
use iii_sdk::protocol::TriggerRequest;
use serde_json::json;

let receipt = worker
    .trigger(TriggerRequest {
        function_id: "email::send".to_string(),
        payload: json!({ "to": "[e-mail]", "subject": "hi" }),
        action: Some(TriggerAction::Enqueue { queue: "email-jobs".to_string() }), // "queue" specifies the name of the queue
        timeout_ms: None,
    })
    .await?;
// receipt["messageReceiptId"] identifies the enqueued job

Pub/Sub Queues

Queues can also be used in a publish/subscribe form when multiple listeners need to subscribe to the same data and it’s important that the messages be durable (ie. will succeed).

Consuming messages

A consumer can bind to a message by registering a Trigger for durable:subscriber trigger to it. The engine runs the function once per message, passing the published data as the payload. Returning normally acknowledges the message; throwing nacks it, so it is retried and eventually dead-lettered.

1. In a worker, register the consumer function and subscribe it to the topic. If you do not have a worker yet, scaffold one with iii worker init, then edit its source:

iii worker init email-worker --language typescript

Node / TypeScript

import { registerWorker } from "iii-sdk";

const url = process.env.III_URL;
if (!url) throw new Error("III_URL must be set");
const worker = registerWorker(url, { workerName: "email-worker" });

// receives the `data` from each published message
worker.registerFunction("email::send", async (msg: { to: string; subject: string }) => {
  // do the work here; throw to nack and let the message retry
  return { sent: true };
});

worker.registerTrigger({
  type: "durable:subscriber",
  function_id: "email::send",
  config: { topic: "emails" },
});

Python

import os
from iii import register_worker, InitOptions

worker = register_worker(
    os.environ["III_URL"],
    InitOptions(worker_name="email-worker"),
)

# receives the `data` from each published message
def send(msg: dict) -> dict:
    # do the work here; raise to nack and let the message retry
    return {"sent": True}

worker.register_function("email::send", send)

worker.register_trigger({
    "type": "durable:subscriber",
    "function_id": "email::send",
    "config": {"topic": "emails"},
})

Rust

use iii_sdk::{InitOptions, RegisterFunction, register_worker};
use iii_sdk::protocol::RegisterTriggerInput;
use serde::Deserialize;
use schemars::JsonSchema;
use serde_json::json;

#[derive(Deserialize, JsonSchema)]
struct Email {
    to: String,
    subject: String,
}

let url = std::env::var("III_URL").expect("III_URL must be set");
let worker = register_worker(&url, InitOptions::default());

// receives the `data` from each published message
worker.register_function("email::send", RegisterFunction::new(|_msg: Email| {
    // do the work here; return an error to nack and let the message retry
    Ok(json!({ "sent": true }))
}));

worker.register_trigger(RegisterTriggerInput {
    trigger_type: "durable:subscriber".into(),
    function_id: "email::send".into(),
    config: json!({ "topic": "emails" }),
    metadata: None,
})?;

2. Add the worker to start it:

iii worker add ./email-worker

Publishing a message

With the consumer running, publish to its topic. The engine delivers the data to every subscriber, so email::send runs once per message:

# publish a message to the "emails" topic
iii trigger iii::durable::publish --json '{"topic":"emails","data":{"to":"[e-mail]","subject":"hi"}}'

Open the console and go to the Traces tab to watch the message flow from the publish through to email::send running.

Retries and delivery

The durable:subscriber trigger’s config controls how each subscriber consumes:

Field	Default	Description
topic	required	Topic to consume. queue is accepted as an alias.
condition_function_id	none	Function invoked with the message before the handler; returning false skips the handler for that message.
queue_config	none	Per-subscriber delivery tuning (below).

A failed delivery retries with exponential backoff (1 second, then 2 seconds) for up to 3 attempts, then the message dead-letters.

queue_config accepts the following fields:

Field	Default	Description
type	concurrent	concurrent processes messages in parallel; fifo processes them strictly one at a time.
concurrency	10	In-flight invocations in concurrent mode. 0 pauses consumption. Ignored by fifo.
maxPriority	none	RabbitMQ adapter only: declares the subscriber’s queue as a priority queue with this many levels (1-255).

For example, to process messages strictly one at a time instead of concurrently, register the trigger with a fifo queue:

Node / TypeScript

worker.registerTrigger({
  type: "durable:subscriber",
  function_id: "email::send",
  config: { topic: "emails", queue_config: { type: "fifo" } },
});

Python

worker.register_trigger({
    "type": "durable:subscriber",
    "function_id": "email::send",
    "config": {"topic": "emails", "queue_config": {"type": "fifo"}},
})

Rust

worker.register_trigger(RegisterTriggerInput {
    trigger_type: "durable:subscriber".into(),
    function_id: "email::send".into(),
    config: json!({ "topic": "emails", "queue_config": { "type": "fifo" } }),
    metadata: None,
})?;

Inspecting Queue Topics

These commands list both kinds of queue. A pub/sub topic appears once a function subscribes to it, and shows broker_type: "builtin". (Publishing to a topic that nothing subscribes to does not register it, so there is nothing to inspect.) A named queue appears as soon as its queue_configs entry is defined, and shows broker_type: "function_queue".

List every topic (this inspects the emails topic from above):

iii trigger engine::queue::list_topics

[{ "name": "emails", "broker_type": "builtin", "subscriber_count": 1 }]

Get stats for the topic (depth is messages waiting for the consumer, dlq_depth is dead-lettered). A topic whose consumer keeps up sits at depth: 0. For a named queue, consumer_count reports its configured concurrency slots (10 for email-jobs):

iii trigger engine::queue::topic_stats topic=emails

{ "depth": 0, "consumer_count": 1, "dlq_depth": 0, "config": null }

Inspecting Dead Letter Queue Messages

A message reaches the dead-letter queue only once its subscribed function exhausts its retries, so the DLQ functions return empty until something fails.

Forcing a message into the dead-letter queue

To see the DLQ populated, make email::send fail by changing the handler to throw. Each message then fails its 3 delivery attempts (a few seconds with the exponential backoff) and dead-letters.

Node / TypeScript

worker.registerFunction("email::send", async () => {
  throw new Error("forced failure");
});

worker.registerTrigger({
  type: "durable:subscriber",
  function_id: "email::send",
  config: { topic: "emails" },
});

Python

def send(_msg: dict) -> dict:
    raise Exception("forced failure")

worker.register_function("email::send", send)

worker.register_trigger({
    "type": "durable:subscriber",
    "function_id": "email::send",
    "config": {"topic": "emails"},
})

Rust

worker.register_function("email::send", RegisterFunction::new(|_msg: serde_json::Value| {
    Err::<serde_json::Value, _>(iii_sdk::Error::Handler("forced failure".into()))
}));

worker.register_trigger(RegisterTriggerInput {
    trigger_type: "durable:subscriber".into(),
    function_id: "email::send".into(),
    config: json!({ "topic": "emails" }),
    metadata: None,
})?;

Now publish a message; after the retries run out it lands in the DLQ (exact ids, timestamps, and sizes vary per run):

iii trigger iii::durable::publish --json '{"topic":"emails","data":{"to":"[e-mail]","subject":"hi"}}'

Listing topics with dead-lettered messages

iii trigger engine::queue::dlq_topics

[{ "topic": "emails", "broker_type": "builtin", "message_count": 1 }]

Browsing dead-lettered messages

iii trigger engine::queue::dlq_messages topic=emails

[
  {
    "id": "0b9c…",
    "payload": { "to": "[e-mail]", "subject": "hi" },
    "error": "ErrorBody { code: \"invocation_failed\", message: \"forced failure\"...",
    "failed_at": 1718900000,
    "retries": 3,
    "size_bytes": 64
  }
]

Redriving dead-lettered messages

Fix the code back to what it was originally, then move the topic’s dead-lettered messages back to the main queue for reprocessing:

iii trigger iii::queue::redrive topic=emails

{ "queue": "emails", "redriven": 1 }

The fixed function now processes them, so the DLQ is empty again:

iii trigger engine::queue::dlq_messages topic=emails

Redriving or discarding a single message

To handle one message instead of the whole topic, pass the id from engine::queue::dlq_messages. Redrive one message back to the main queue:

iii trigger iii::queue::redrive_message topic=emails message_id=0b9c…

{ "queue": "emails", "message_id": "0b9c…", "redriven": 1 }

Or discard it, deleting it from the DLQ permanently:

iii trigger iii::queue::discard_message topic=emails message_id=0b9c…

{ "queue": "emails", "message_id": "0b9c…", "redriven": 1 }