Queue Monitor

The Strata Queue Monitor (SQM) is a Strata service that captures messages flowing through your system and writes them to one or more output backends -- Elasticsearch, Logstash, or the console. It gives you full visibility into every request and response, making it an essential debugging and observability tool.

SQM is safe to run in production. It processes messages on its own queue, scales like any other Strata service, and has no impact on the critical path of your services.

How It Works

SQM is a normal Strata service. It listens on the StrataQueueMonitor service group and exposes a single operation: monitor.logMessage. This operation accepts a queue and an envelope (a Strata request or response envelope) and writes it to whichever output plugins are enabled.

Because SQM is just a service, it gets the same guarantees as everything else in Strata -- message acknowledgement, replay on crash, configurable queue length, and horizontal scaling.

Sending Messages to SQM

You can send messages to SQM manually, but the intended approach is the message-logging middleware. Add it to any service and it automatically forwards every incoming request and outgoing response to SQM without blocking the service's normal processing.

The middleware sends messages fire-and-forget. If SQM is down, a warning is logged but service processing continues unaffected.

Installation

SQM is deployed as a Docker container. It is not published as an npm package.

docker run -d --name sqm \
    -v /path/to/config/config.yml:/app/config/config.yml \
    stratajs/strata-queue-monitor:latest

Or in a compose.yaml:

services:
    sqm:
        image: stratajs/strata-queue-monitor:latest
        restart: unless-stopped
        volumes:
            - ./config:/app/config:ro

docker compose up -d

Configuration

SQM uses a config.yml file. Copy the default from the Docker image as a starting point:

docker cp sqm:/app/config/config.yml ./config.yml

The config file supports environment variable substitution using ${ MY_VAR } syntax. Variables are replaced before YAML parsing, so they are substituted verbatim.

Service and Backend

service:
    serviceGroup: 'StrataQueueMonitor'

backend:
    type: 'redis'
    redis:
        host: 'localhost'
        port: 6379
        # username: ${ REDIS_USERNAME }
        # password: ${ REDIS_PASSWORD }

The backend section is a standard Strata backend config. The redis key is passed directly to ioredis.

Processing

processing:
    auth:
        # Decode JWT auth tokens into a separate key in the logged message
        decode: 'jwt'
        # Key name for the decoded token (default: 'authDecode')
        key: 'token'
        # Replace the original auth value with 'SCRUBBED'
        scrub: true

The processing.auth section controls how the auth field on requests is handled before writing to the output. This is useful for decoding JWTs into readable form or scrubbing tokens from logs.

Logging

logging:
    level: 'info'       # trace | debug | info | warn | error | critical
    prettyPrint: false   # Enable pino-pretty (development only)

Output Plugins

SQM supports multiple output plugins. Enable whichever ones you need -- multiple can be active at the same time.

outputConfigs:

    # Elasticsearch (direct)
    elastic:
        enabled: true
        forceRefresh: false
        indexPrefix: 'strata-queue-monitor'
        bulkConfig:
            enabled: true
            maxMessages: 20
            flushSize: 1000000       # bytes
            flushInterval: 3000      # ms
            concurrency: 4
        clientOptions:
            nodes:
                - http://localhost:9200
            # auth:
            #     username: ${ ELASTIC_USERNAME }
            #     password: ${ ELASTIC_PASSWORD }

    # Logstash (HTTP input)
    logstashHTTP:
        enabled: false
        url: http://localhost:5555

    # Console (stdout)
    console:
        enabled: false

    # No-op (discard all messages)
    none:
        enabled: false

Elasticsearch Plugin

Option	Type	Description
enabled	boolean	Enable this plugin.
forceRefresh	boolean	Force Elasticsearch to refresh the index after each insert. Useful for development, not recommended in production.
indexPrefix	string	Prefix for the Elasticsearch index name. Indices are created as <prefix>-YYYY.MM.DD.
bulkConfig.enabled	boolean	Use the bulk API for writes. Significantly improves throughput.
bulkConfig.maxMessages	number	Number of messages to buffer before flushing.
bulkConfig.flushSize	number	Maximum buffer size in bytes before flushing.
bulkConfig.flushInterval	number	Maximum time in ms to hold messages before flushing, even if flushSize is not reached.
bulkConfig.concurrency	number	Concurrent bulk requests to Elasticsearch.
clientOptions	object	Passed directly to @elastic/elasticsearch. Supports all options from that library.

Logstash HTTP Plugin

Option	Type	Description
enabled	boolean	Enable this plugin.
url	string	URL of the Logstash HTTP input endpoint.

Console Plugin

Writes messages to stdout. Useful for development or piping to external log aggregators.

None Plugin

Discards all messages. Useful if you want SQM running (consuming from the queue) but don't need to store anything.

Kibana Setup

Once messages are flowing to Elasticsearch, connect Kibana to view them:

1. Open Kibana (default: http://localhost:5601).
2. Navigate to Stack Management > Index Patterns.
3. Create an index pattern: strata-queue-monitor-*.
4. Go to Discover, select the strata-queue-monitor-* pattern, and browse your messages.

Each document contains fields like context, operation, status, request, response, timestamps, and client/ service identifiers. The full shape is documented in the source under StrataQueueMonitorMessage.

Development

To develop SQM locally, you need an Elasticsearch cluster and optionally Kibana and Logstash. The repository includes a compose file that sets up a 3-node Elasticsearch cluster, Kibana, and Logstash:

docker compose -p strata-dev -f compose/dev-env.yml up -d

This starts:

• Elasticsearch on http://localhost:9200 (3 nodes)
• Kibana on http://localhost:5601
• Logstash on http://localhost:5555 (HTTP input) and http://localhost:9600 (monitoring)

Then run SQM locally:

npm install
npm start

The default config.yml is already configured to connect to the local Elasticsearch cluster. Enable the console output plugin for quick feedback during development.