Skip to main content

Running​

The simpliest way to run preq and the latest community CREs is using standard input. CREs will be automatically updated as new releases are available.

Try our demo application​

Once you've installed preq, let's give it a try using our demo application. This will help you quickly see how things work before pointing it at your data.

Try the following command.

curl -s https://docs.prequel.dev/demo/application.log | preq

You should see that a problem (cre-2024-0007) was detected:

Parsing rules           done! [3 rules in 3ms; 433 rules/s]
Problems detected       done! [1 in 7ms; 144/s]
Reading stdin           done! [208.64KB in 4ms; 53.01MB/s]
Matching lines          done! [1.01K lines in 4ms; 275.29K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T10:00:19-04:00]

Great! Run with preq -n - for a detailed report. Or see this demo on the playground to learn how to write a CRE.

Now you're ready to try preq on your own data!

Now try your data​

Here are some examples.

kubectl logs -l env=prod | preq

CRE Reports​

preq prints CRE detections to standard out. It also records details for each detection in a JSON report. The concise details on standard out list the CRE ID, its severity, the number of hits in the data for the CRE, and the first observed timestamp.

FieldDescription
creIdCRE identification number for this problem
severityNumeric severity level. See Severity.

Example report​

The report provides additional context on the problem, the impact, its mitigation, and references. It also records the specific matches in the data by the CRE rule with their timestamps.

preq-report.json
[
  {
    "cre": {
      "id": "CRE-2024-0007",
      "title": "RabbitMQ Mnesia overloaded recovering persistent queues",
      "category": "message-queue-problems",
      "tags": [
        "cre-2024-0007",
        "known-problem",
        "rabbitmq"
      ],
      "author": "Prequel",
      "description": "- The RabbitMQ cluster is processing a large number of persistent mirrored queues at boot. \n",
      "impact": "- RabbitMQ is unable to process any new messages and can cause outages in consumers and producers.\n",
      "cause": "- The Erlang process, Mnesia, is overloaded while recovering persistent queues on boot. \n",
      "mitigation": "- Adjusting mirroring policies to limit the number of mirrored queues\n- Remove high-availability policies from queues\n- Add additional CPU resources and restart the RabbitMQ cluster\n- Use [lazy queues](https://www.rabbitmq.com/docs/lazy-queues) to avoid incurring the costs of writing data to disk \n",
      "references": [
        "https://groups.google.com/g/rabbitmq-users/c/ekV9tTBRZms/m/1EXw-ruuBQAJ"
      ],
      "applications": [
        {
          "name": "rabbitmq",
          "version": "3.9.x"
        }
      ]
    },
    "hits": [
      {
        "timestamp": "2025-03-11T09:00:19-05:00",
        "entry": "2025-03-11 14:00:19.421865+00:00 [erro] \u003c0.229.0\u003e Discarding message {'$gen_cast',{force_event_refresh,#Ref\u003c0.449530684.1179910147.46753\u003e}} from \u003c0.229.0\u003e to \u003c0.3159.0\u003e in an old incarnation (1741605434) of this node (1741701615) \u003cA\u003e"
      },
      {
        "timestamp": "2025-03-11T09:00:22-05:00",
        "entry": "2025-03-11 14:00:20.144956+00:00 [warn] \u003c0.247.0\u003e Mnesia('rabbit@rabbitmq-0.svc.cluster.local'): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}"
      },
      {
        "timestamp": "2025-03-11T09:00:20-05:00",
        "entry": "2025-03-11 14:00:19.421872+00:00 [erro] \u003c0.229.0\u003e Discarding message {'$gen_cast',{force_event_refresh,#Ref\u003c0.449530684.1179910147.46753\u003e}} from \u003c0.229.0\u003e to \u003c0.3156.0\u003e in an old incarnation (1741605434) of this node (1741701615)"
      },
      {
        "timestamp": "2025-03-11T09:00:23-05:00",
        "entry": "2025-03-11 14:00:20.177194+00:00 [warn] \u003c0.247.0\u003e Mnesia('rabbit@rabbitmq-0.svc.cluster.local'): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}"
      }
    ],
    "id": "CRE-2024-0007",
    "rule_hash": "",
    "rule_id": "5UD1RZxGC5LJQnVpAkV11A",
    "timestamp": "2025-03-11T09:00:19-05:00"
  }
]

See the CRE specification for more details.

Custom report name​

The report will be saved to preq-report-<timestamp-epoch>.json unless -n is specified to change the name of the report.

cat /var/log/rabbitmq.log | preq -n myreport.json

Example output

Parsing rules           done! [1 rules in 1ms; 497 rules/s]
Problems detected       done! [1 in 2ms; 494/s]
Reading stdin           done! [2.88KB in 1ms; 1.97MB/s]
Matching lines          done! [14 lines in 1ms; 9.57K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to myreport.json

Skip generating a report​

Use -n "" to avoid generating a report file. This is useful during development.

cat /var/log/rabbitmq.log | preq -n ""

Example output

Parsing rules           done! [1 rules in 1ms; 497 rules/s]
Problems detected       done! [1 in 2ms; 494/s]
Reading stdin           done! [2.88KB in 1ms; 1.97MB/s]
Matching lines          done! [14 lines in 1ms; 9.57K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Send report to standard out​

Use -n "-" to send the report to standard out instead of the filesystem.

cat ./examples/02-example.log | preq -r ./examples/02-set-multiple-example-good-window.yaml -d -n -

Example output

Parsing rules           done! [1 rules in 1ms; 923 rules/s]
Problems detected       done! [1 in 1ms; 915/s]
Reading stdin           done! [822B in 0s; 4.92MB/s]
set-example-2        critical [1 hits @ 2019-02-05T06:07:39-06:00]
Matching lines          done! [10 lines in 0s; 57.80K lines/s]
[
  {
    "cre": {
      "id": "set-example-2"
    },
    "hits": [
      {
        "timestamp": "2019-02-05T06:07:39-06:00",
        "entry": "2019/02/05 12:07:39 [emerg] 1655#1655: bind() to test"
      },
      {
        "timestamp": "2019-02-05T06:07:38-06:00",
        "entry": "2019/02/05 12:07:38 [emerg] 1655#1655: bind() to foo bar"
      },
      {
        "timestamp": "2019-02-05T06:07:43-06:00",
        "entry": "2019/02/05 12:07:43 [emerg] 1655#1655: still could not bind() to baaaz"
      }
    ],
    "id": "set-example-2",
    "rule_hash": "",
    "rule_id": "",
    "timestamp": "2019-02-05T06:07:39-06:00"
  }
]

Silent mode​

Use -q to stop printing progress and CRE summaries to standard out.

cat ./examples/02-example.log | preq -r ./examples/02-set-multiple-example-good-window.yaml -d -q -n -

Example output

[
  {
    "cre": {
      "id": "set-example-2"
    },
    "hits": [
      {
        "timestamp": "2019-02-05T06:07:39-06:00",
        "entry": "2019/02/05 12:07:39 [emerg] 1655#1655: bind() to test"
      },
      {
        "timestamp": "2019-02-05T06:07:38-06:00",
        "entry": "2019/02/05 12:07:38 [emerg] 1655#1655: bind() to foo bar"
      },
      {
        "timestamp": "2019-02-05T06:07:43-06:00",
        "entry": "2019/02/05 12:07:43 [emerg] 1655#1655: still could not bind() to baaaz"
      }
    ],
    "id": "set-example-2",
    "rule_hash": "",
    "rule_id": "",
    "timestamp": "2019-02-05T06:07:39-06:00"
  }
]

Debug logs​

Use -l <LEVEL> to print debug logs at the info, debug, trace, error, or warn level. Logs are sent to standard error.

cat /var/log/rabbitmq.log | preq -r ~/rule.yaml -l error

Example output

Apr  2 23:25:04.512448 ERR engine.go:207 > Duplicate rule hash id. Aborting... id=5UD1RZxGC5LJQnVpAkV11A
Apr  2 23:25:04.512531 ERR engine.go:463 > Failed to load rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Apr  2 23:25:04.512555 ERR engine.go:491 > Failed to compile rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Apr  2 23:25:04.512569 ERR preq.go:231 > Failed to load rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Rules error: duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007

Add -j to print the debug logs in JSON format.

preq -s nope -l error -j

Example output

{"level":"error","error":"open nope: no such file or directory","time":1744173105369186,"caller":"preq.go:81","message":"Failed to parse data sources file"}
{"level":"error","error":"open nope: no such file or directory","time":1744173105369215,"caller":"preq.go:198","message":"Failed to parse data sources"}
Data error: open nope: no such file or directory

Rules​

preq will download and run the latest CRE rules from the community.

Developing rules​

Use -r to run another rule document in addition to the community CRE rules. Ensure the rules do not contain duplicate IDs, like CRE ID, rule ID, or rule hash.

cat /var/log/rabbitmq.log | preq -r ~/my-new-rules.yaml

Example output

Parsing rules           done! [1 rules in 0s; 649 rules/s]
Problems detected       done! [1 in 1ms; 643/s]
Reading stdin           done! [2.88KB in 0s; 2.50MB/s]
Matching lines          done! [14 lines in 0s; 12.20K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to preq-report-1743654939.json

Use -d to disable running community CREs while developing a new rule.

Accept updates​

Use -y to avoid interactive input prompts when new CRE or preq updates are available for download.

cat /var/log/rabbitmq.log | preq -r ~/rule.yaml -d -y

Example output

package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz
Downloading update       ... done! [4.19KB in 0s; 37.77MB/s]
package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz.sha256
package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz.sig
ECDSA signature and sha256 hash verified
Parsing rules           done! [1 rules in 1ms; 713 rules/s]
Problems detected       done! [1 in 1ms; 706/s]
Reading stdin           done! [2.88KB in 0s; 3.42MB/s]
Matching lines          done! [14 lines in 0s; 16.75K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to preq-report-1743655171.json

Stop time​

By default preq will replay and evaluate detections until the end of time in the input data. Use -e to stop detecting CREs in the input data at the specified time.

cat /var/log/rabbitmq.log | preq -r ~/rule.yaml -d -e "2025-03-11T14:00:20Z"

Example output

Parsing rules           done! [1 rules in 1ms; 565 rules/s]
Problems detected       done! [0 in 1ms; 0/s]
Reading stdin           done! [2.88KB in 1ms; 2.39MB/s]
Matching lines          done! [2 lines in 1ms; 1.66K lines/s]

Wrote report to preq-report-1743655312.json

Data sources​

Use -s to provide a data sources configuration file. See Data Sources for more information.

preq -s ./examples/40-sources.yaml

Example output

Parsing rules           done! [3 rules in 1ms; 14 rules/s]
Problems detected       done! [0 in 1m24.687s; 0/s]
Reading my-gke-metrics  done! [28.01GB in 1m24.685s; 330.79MB/s]
Matching lines          done! [114.42M lines in 1m24.685s; 1.35M lines/s]

Wrote report to preq-report-1743656723.json

Timestamp Regex and Format​

Use -x and -t to specify a regular expression and a format string to parse custom timestamp formats. See custom timestamp formats for more information.

Command line syntax​

preq -h

Example output

Usage: preq [flags]

Flags:
  -h, --help                  Show context-sensitive help.
  -d, --disabled              Do not run community CREs
  -e, --stop=STRING           Stop time
  -j, --json-logs             Print logs in JSON format to stderr
  -k, --skip=10               Skip the first N lines for timestamp detection
  -l, --level=STRING          Print logs at this level to stderr
  -n, --report-file=STRING    Report filename
  -q, --quiet                 Quiet mode, do not print progress
  -r, --rules=STRING          Path to a CRE file
  -s, --source=STRING         Path to a data source file
  -t, --format=STRING         Format to use for timestamps
  -v, --version               Print version and exit
  -w, --window=STRING         Reorder lookback window duration
  -x, --regex=STRING          Regex to match for extracting timestamps
  -y, --accept-updates        Accept updates to rules or new release