Skip to main content

Running​

The simpliest way to run preq and the latest community CREs is using standard input. CREs are frequently (and automatically) updated as new releases are available.

Once you've installed preq, let's give it a try on a simple demo service with a few problems. We'll use a simple demo application.

Step 1: Install and run the demo service​

The demo service executable is available for all supported operating systems and architectures. You can also run it directly with Golang v1.24.1.

curl -sL "$(curl -s https://api.github.com/repos/prequel-dev/preq-demo-app/releases/latest \
  | jq -r '.assets[] | select(.name | test("demo-linux-amd64$")) .browser_download_url')" \
  -o demo && chmod +x demo && ./demo > preq-demo.log 2>&1

Step 2: Trigger a problem​

In a new Terminal window, run this command to trigger the problem:

curl http://localhost:8080/panic

Example output:

{"status":"goroutine panic triggered"}

This causes the server to crash. You may also get this error instead:

curl: (52) Empty reply from server

Step 3: Detect the problem​

cat preq-demo.log | preq -o -

You should see that CRE-2025-0918 was detected:

Parsing rules           done! [3 rules in 3ms; 433 rules/s]
Problems detected       done! [1 in 7ms; 144/s]
Reading stdin           done! [208.64KB in 4ms; 53.01MB/s]
Matching lines          done! [1.01K lines in 4ms; 275.29K lines/s]
CRE-2025-0918        critical [1 hits @ 2025-03-11T10:00:19-04:00]
[
  {
    "cre": {
      "id": "CRE-2025-0918",
      "severity": 0,
      "title": "Demo Application Crashing due to ENV Misconfiguration",
      "category": "demo-problems",
      "tags": [
        "demo-problem"
      ],
      "author": "Tony Meehan",
      "description": "- The author of the demo application thought it would be interesting to make the application break if a secret environment variable was not set.\n",
      "impact": "- The application crashes and users can no longer access the demo service.\n",
      "cause": "- It's a demo application that needs to have PANIC=off set in the environment.\n",
      "mitigation": "Run the demo service with PANIC=off\n\n```\nPANIC=off ./demo\n```\n",
      "references": [
        "https://docs.prequel.dev"
      ],
      "applications": [
        {
          "name": "preq-demo-app"
        }
      ]
    },
    "hits": [
      {
        "timestamp": "2025-06-03T23:16:32-05:00",
        "entry": "2025/06/04 04:16:32 level=info msg=\"starting server\" addr=:8080panic: intentional panic inside goroutine for demo purposesgoroutine 104 [running]:main.panicHandler.func1()\t/src/cmd/demo.go:102 +0x25created by main.panicHandler in goroutine 102\t/src/cmd/demo.go:101 +0x31"
      },
      {
        "timestamp": "2025-06-03T23:16:32-05:00",
        "entry": "2025/06/04 04:16:32 level=info msg=\"starting server\" addr=:8080panic: intentional panic inside goroutine for demo purposesgoroutine 104 [running]:main.panicHandler.func1()\t/src/cmd/demo.go:102 +0x25created by main.panicHandler in goroutine 102\t/src/cmd/demo.go:101 +0x31"
      }
    ],
    "id": "CRE-2025-0918",
    "rule_hash": "7VLbMVpbFTFPXEwjBprnx1U2qAuK8pHvWeSJmt7U3ss5",
    "rule_id": "YJ2r3QPQDj8iFnXmoUSG8p",
    "timestamp": "2025-06-03T23:16:32-05:00"
  }
]

Step 4: Mitigation​

The rule provides a mitigation for the demo problem. Start the service with the mitigation applied from the mitigation field above and verify the problem no longer occurs.

curl -sL "$(curl -s https://api.github.com/repos/prequel-dev/preq-demo-app/releases/latest \
  | jq -r '.assets[] | select(.name | test("demo-linux-amd64$")) .browser_download_url')" \
  -o demo && chmod +x demo && PANIC=off ./demo > preq-demo.log 2>&1

Step 5: Try to trigger the problem again​

In a new Terminal window, run this command to try to trigger the problem:

curl http://localhost:8080/panic

Example output:

{"status":"panic disabled"}

Step 6: Validate the problem is fixed​

cat preq-demo.log | preq -o -

You should see that CRE-2025-0918 is no longer detected:

Parsing rules           done! [3 rules in 3ms; 433 rules/s]
Problems detected       done! [1 in 7ms; 144/s]
Reading stdin           done! [208.64KB in 4ms; 53.01MB/s]
Matching lines          done! [1.01K lines in 4ms; 275.29K lines/s]

Congrats! Now let's write a new rule.

Write a rule​

Now let's write a new rule for a new problem in this service. Let's first trigger the new problem.

Step 1: Trigger a new problem​

In a new Terminal window, run this command to trigger the problem:

curl http://localhost:8080/migrate

Example output:

migration failed

Step 2: Create a rule in the CRE Playground​

Look at the logs of the demo service.

cat preq-demo.log

Example output:

2025/06/04 00:20:12 level=info msg="configuration" panic_mode=true
2025/06/04 00:20:12 level=info msg="starting server" addr=:8080
2025/06/04 00:20:20 level=info msg="running migration"
2025/06/04 00:20:20 level=error msg="migration failed" err=alter table: SQL logic error: no such table: imaginary (1)
2025/06/04 00:20:20 level=info method=GET path=/migrate status=500 duration=519.466µs

Copy the log output to the CRE Playground. Click this link to a playground with the data already copied for you.

Add a new match term to match "migration failed":

custom-rule.yaml
rules:
  - cre:
      id: CRE-2025-0919
      severity: 0
      title: My First Custom Rule
      category: demo-problems
      author: My Name
      description: |
        - This is my first custom rule
    metadata:
      id: 6qHwZYugnnmUoB7FwVJdm7
    rule:
      set:
        event:
          source: cre.log.demo
        match:
          - "migration failed"

We wanted to get you started with a simple rule but this is just the beginning!

The rule syntax supports more advanced features such as:

  • regular expressions if you want to match other parts of the log line
  • sets for rules that require more than one condition
  • negative conditions if you want to reduce noise or false positives
  • sequences if you care about the order of the conditions
  • nested conditions for complex combinations of positive and negative conditions

See Syntax Reference for more details.

Step 3: Validate the rule​

Click "Test Rule" to validate your new rule works after you made the changes above.

Step 4: Use the rule​

Now save the rule from the playground in a new file called custom-rule.yaml.

cat <<'EOF' > custom-rule.yaml
rules:
  - cre:
      id: CRE-2025-0919
      severity: 0
      title: My First Custom Rule
      category: demo-problems
      author: My Name
      description: |
        - This is my first custom rule
    metadata:
      id: 6qHwZYugnnmUoB7FwVJdm7
    rule:
      set:
        event:
          source: cre.log.demo
        match:
          - "migration failed"
EOF
cat preq-demo.log | preq -d -r ./custom-rule.yaml

You should see that CRE-2025-0919 was detected:

Parsing rules           done! [3 rules in 3ms; 433 rules/s]
Problems detected       done! [1 in 7ms; 144/s]
Reading stdin           done! [208.64KB in 4ms; 53.01MB/s]
Matching lines          done! [1.01K lines in 4ms; 275.29K lines/s]
CRE-2025-0919        critical [1 hits @ 2025-03-11T10:00:19-04:00]

Automated Runbooks​

You can run automated actions on new detections using preq -a <actions.yaml>.

Send detections to Slack​

Create an actions.yaml file.

actions.yaml
actions:
  - type: slack
    regex: "CRE*"
    slack:
      webhook_url: https://hooks.slack.com/services/<webhook>
      message_template: |
           *preq detection*: [{{ field .cre "Id" }}] {{ field .cre "Title" }}
           
           {{ (index .hits 0).Timestamp }}: {{ (index .hits 0).Entry }}

Templates are used to reference fields in the CRE reports. You can configure those in this file. The regex field can be used to route specific CRE rules to specific actions using a regular expression on the CRE ID.

Then run preq:

cat preq-demo.log | preq -d -r ./custom-rule.yaml -a actions.yaml

You should see a new Slack notification from preq.

Run an runbook executable​

You can pass the CRE report on standard input to an executable, such as a bash script.

Create a simple runbook bash script called action.sh:

#!/usr/bin/env bash
read -r -d '' EVENT_JSON

echo "Full CRE JSON:"
echo "$EVENT_JSON" | jq .

echo "CRE       = $1"
echo "# of HITS = $2"

# take whatever action is appropriate

Give the new script executable permissions:

chmod +x action.sh

Then create an actions.yaml file:

actions.yaml
actions:
  - type: exec
    regex: "CRE*"
    exec:
      path: ./action.sh
      args:
        - '{{ field .cre "Id" }}'
        - '{{ len .hits }}'

Then run preq with -a:

cat preq-demo.log | preq -d -r ./custom-rule.yaml -a actions.yaml

Example output:

Parsing rules           done! [1 rules in 0s; 1.45K rules/s]
Problems detected       done! [1 in 1ms; 1.40K/s]
Reading stdin           done! [390B in 0s; 1.51MB/s]
CRE-2025-0919        critical [1 hits @ 2025-06-04T04:19:33-05:00]
Matching lines          done! [5 lines in 0s; 18.73K lines/s]
Full CRE JSON:
{
  "cre": {
    "id": "CRE-2025-0919",
    "severity": 0,
    "title": "My First Custom Rule",
    "category": "demo-problems",
    "author": "My Name",
    "description": "- This is my first custom rule\n"
  },
  "hits": [
    {
      "timestamp": "2025-06-04T04:19:33-05:00",
      "entry": "2025/06/04 09:19:33 level=error msg=\"migration failed\" err=alter table: SQL logic error: no such table: imaginary (1)"
    }
  ],
  "id": "CRE-2025-0919",
  "rule_hash": "J2xPvAziSSpe3d62BCf9FPquAKJsvcRGxMZTP7ND2P1m",
  "rule_id": "6qHwZYugnnmUoB7FwVJdm7",
  "timestamp": "2025-06-04T04:19:33-05:00"
}
CRE       = CRE-2025-0919
# of HITS = 1

Create a new JIRA issue​

You can automatically create a JIRA issue when a new detection occurs.

actions.yaml
actions:
  - type: jira
    regex: "CRE*"
    jira:
      project_key: KAN
      webhook_url: https://<YOUR-TEAM>.atlassian.net/rest/api/3/issue
      secret_env: JIRA_TOKEN
      summary_template: |
        *preq detection*: [{{ field .cre "Id" }}] {{ field .cre "Title" }}
      description_template: |
        {{ (index .hits 0).Timestamp }}: {{ (index .hits 0).Entry }}

Export your JIRA token to an environment variable named JIRA_TOKEN.

Scheduled Jobs​

You can schedule preq to run in a cronjob (or Kubernetes cronjob). When combined with an automated runbook, preq can be used to detect, notify, and remediate problems without sending data anywhere.

Use preq -j to generate a Kubernetes cronjob.

preq -j

Expected output:

Cronjob template written to cronjob.yaml

Customize the cronjob Yaml to add your automated runbook action and to target services.

Youc an also use crontab -e to run preq regularly with a data sources template outside of Kubernetes:

crontab
*/15 * * * *  /path/to/bin/preq -s /path/to/data-sources.yaml -n /path/to/cre/reports/cre-report-`date +\%Y\%m\%d\%H\%M\%S`.json

Additional Examples​

Now check out these additional examples. Or learn how to set up scheduled jobs to run preq automatically and send notifications for new detections.

Try your data​

Congrats, you're ready to try preq on your own data! First start with piping data to preq via stdin. Here are some examples.

kubectl preq pg17-postgresql-0

Once you know which data sources you want to regularly test, generate a data source template and set up a scheduled job.

CRE reports​

preq prints CRE detections to standard out. It also records details for each detection in a JSON report. The concise details on standard out list the CRE ID, its severity, the number of hits in the data for the CRE, and the first observed timestamp. Check out the CRE schema for more details.

The report provides additional context on the problem, the impact, its mitigation, and references. It also records the specific matches in the data by the CRE rule with their timestamps.

preq-report.json
[
  {
    "cre": {
      "id": "CRE-2024-0007",
      "title": "RabbitMQ Mnesia overloaded recovering persistent queues",
      "category": "message-queue-problems",
      "tags": [
        "cre-2024-0007",
        "known-problem",
        "rabbitmq"
      ],
      "author": "Prequel",
      "description": "- The RabbitMQ cluster is processing a large number of persistent mirrored queues at boot. \n",
      "impact": "- RabbitMQ is unable to process any new messages and can cause outages in consumers and producers.\n",
      "cause": "- The Erlang process, Mnesia, is overloaded while recovering persistent queues on boot. \n",
      "mitigation": "- Adjusting mirroring policies to limit the number of mirrored queues\n- Remove high-availability policies from queues\n- Add additional CPU resources and restart the RabbitMQ cluster\n- Use [lazy queues](https://www.rabbitmq.com/docs/lazy-queues) to avoid incurring the costs of writing data to disk \n",
      "references": [
        "https://groups.google.com/g/rabbitmq-users/c/ekV9tTBRZms/m/1EXw-ruuBQAJ"
      ],
      "applications": [
        {
          "name": "rabbitmq",
          "version": "3.9.x"
        }
      ]
    },
    "hits": [
      {
        "timestamp": "2025-03-11T09:00:19-05:00",
        "entry": "2025-03-11 14:00:19.421865+00:00 [erro] \u003c0.229.0\u003e Discarding message {'$gen_cast',{force_event_refresh,#Ref\u003c0.449530684.1179910147.46753\u003e}} from \u003c0.229.0\u003e to \u003c0.3159.0\u003e in an old incarnation (1741605434) of this node (1741701615) \u003cA\u003e"
      },
      {
        "timestamp": "2025-03-11T09:00:22-05:00",
        "entry": "2025-03-11 14:00:20.144956+00:00 [warn] \u003c0.247.0\u003e Mnesia('rabbit@rabbitmq-0.svc.cluster.local'): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}"
      },
      {
        "timestamp": "2025-03-11T09:00:20-05:00",
        "entry": "2025-03-11 14:00:19.421872+00:00 [erro] \u003c0.229.0\u003e Discarding message {'$gen_cast',{force_event_refresh,#Ref\u003c0.449530684.1179910147.46753\u003e}} from \u003c0.229.0\u003e to \u003c0.3156.0\u003e in an old incarnation (1741605434) of this node (1741701615)"
      },
      {
        "timestamp": "2025-03-11T09:00:23-05:00",
        "entry": "2025-03-11 14:00:20.177194+00:00 [warn] \u003c0.247.0\u003e Mnesia('rabbit@rabbitmq-0.svc.cluster.local'): ** WARNING ** Mnesia is overloaded: {dump_log,write_threshold}"
      }
    ],
    "id": "CRE-2024-0007",
    "rule_hash": "",
    "rule_id": "5UD1RZxGC5LJQnVpAkV11A",
    "timestamp": "2025-03-11T09:00:19-05:00"
  }
]

Command Options​

Custom report name​

The report will be saved to preq-report-<timestamp-epoch>.json unless -o is specified to change the name of the report.

cat /var/log/rabbitmq.log | preq -o myreport.json

Example output

Parsing rules           done! [1 rules in 1ms; 497 rules/s]
Problems detected       done! [1 in 2ms; 494/s]
Reading stdin           done! [2.88KB in 1ms; 1.97MB/s]
Matching lines          done! [14 lines in 1ms; 9.57K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to myreport.json

Skip generating a report​

Use -o "" to avoid generating a report file. This is useful during development.

cat /var/log/rabbitmq.log | preq -o ""

Example output

Parsing rules           done! [1 rules in 1ms; 497 rules/s]
Problems detected       done! [1 in 2ms; 494/s]
Reading stdin           done! [2.88KB in 1ms; 1.97MB/s]
Matching lines          done! [14 lines in 1ms; 9.57K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Send report to standard out​

Use -o "-" to send the report to standard out instead of the filesystem.

cat ./examples/02-example.log | preq -r ./examples/02-set-multiple-example-good-window.yaml -d -o -

Example output

Parsing rules           done! [1 rules in 1ms; 923 rules/s]
Problems detected       done! [1 in 1ms; 915/s]
Reading stdin           done! [822B in 0s; 4.92MB/s]
set-example-2        critical [1 hits @ 2019-02-05T06:07:39-06:00]
Matching lines          done! [10 lines in 0s; 57.80K lines/s]
[
  {
    "cre": {
      "id": "set-example-2"
    },
    "hits": [
      {
        "timestamp": "2019-02-05T06:07:39-06:00",
        "entry": "2019/02/05 12:07:39 [emerg] 1655#1655: bind() to test"
      },
      {
        "timestamp": "2019-02-05T06:07:38-06:00",
        "entry": "2019/02/05 12:07:38 [emerg] 1655#1655: bind() to foo bar"
      },
      {
        "timestamp": "2019-02-05T06:07:43-06:00",
        "entry": "2019/02/05 12:07:43 [emerg] 1655#1655: still could not bind() to baaaz"
      }
    ],
    "id": "set-example-2",
    "rule_hash": "",
    "rule_id": "",
    "timestamp": "2019-02-05T06:07:39-06:00"
  }
]

Silent mode​

Use -q to stop printing progress and CRE summaries to standard out.

cat ./examples/02-example.log | preq -r ./examples/02-set-multiple-example-good-window.yaml -d -q -o -

Example output

[
  {
    "cre": {
      "id": "set-example-2"
    },
    "hits": [
      {
        "timestamp": "2019-02-05T06:07:39-06:00",
        "entry": "2019/02/05 12:07:39 [emerg] 1655#1655: bind() to test"
      },
      {
        "timestamp": "2019-02-05T06:07:38-06:00",
        "entry": "2019/02/05 12:07:38 [emerg] 1655#1655: bind() to foo bar"
      },
      {
        "timestamp": "2019-02-05T06:07:43-06:00",
        "entry": "2019/02/05 12:07:43 [emerg] 1655#1655: still could not bind() to baaaz"
      }
    ],
    "id": "set-example-2",
    "rule_hash": "",
    "rule_id": "",
    "timestamp": "2019-02-05T06:07:39-06:00"
  }
]

Debug logs​

Use -l <LEVEL> to print debug logs at the info, debug, trace, error, or warn level. Logs are sent to standard error.

cat /var/log/rabbitmq.log | preq -r ~/rule.yaml -l error

Example output

Apr  2 23:25:04.512448 ERR engine.go:207 > Duplicate rule hash id. Aborting... id=5UD1RZxGC5LJQnVpAkV11A
Apr  2 23:25:04.512531 ERR engine.go:463 > Failed to load rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Apr  2 23:25:04.512555 ERR engine.go:491 > Failed to compile rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Apr  2 23:25:04.512569 ERR preq.go:231 > Failed to load rules error="duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007"
Rules error: duplicate rule hash id=5UD1RZxGC5LJQnVpAkV11A cre=CRE-2024-007

Custom rules​

A key feature of preq is receiving automatic updates of the latest CRE rules from the community. You can also add custom rules.

Use -r to run another rule document in addition to the community CRE rules. Ensure the rules do not contain duplicate IDs, like CRE ID, rule ID, or rule hash.

cat /var/log/rabbitmq.log | preq -r ~/my-new-rules.yaml

Example output

Parsing rules           done! [1 rules in 0s; 649 rules/s]
Problems detected       done! [1 in 1ms; 643/s]
Reading stdin           done! [2.88KB in 0s; 2.50MB/s]
Matching lines          done! [14 lines in 0s; 12.20K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to preq-report-1743654939.json

Use -d to disable running community CREs while developing a new rule.

Accept updates​

Use -y to avoid interactive input prompts when new CRE or preq updates are available for download.

cat /var/log/rabbitmq.log | preq -r ~/rule.yaml -d -y

Example output

package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz
Downloading update       ... done! [4.19KB in 0s; 37.77MB/s]
package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz.sha256
package name: preq-public-rules.0.3.5.7eda0f45.yaml.gz.sig
ECDSA signature and sha256 hash verified
Parsing rules           done! [1 rules in 1ms; 713 rules/s]
Problems detected       done! [1 in 1ms; 706/s]
Reading stdin           done! [2.88KB in 0s; 3.42MB/s]
Matching lines          done! [14 lines in 0s; 16.75K lines/s]
CRE-2024-0007        critical [2 hits @ 2025-03-11T09:00:19-05:00]

Wrote report to preq-report-1743655171.json

Generate data source template​

Use -g to generate a data source template from your installed CRE rules package.

$ preq -g

Example output

Wrote data source template to data-sources-0.3.12.yaml

Edit the template to point the data sources to the locations of the logs on your system. See Data Sources for more information.

Use a data source template​

Use -s to provide a data sources configuration file.

preq -s ./examples/40-sources.yaml

Example output

Parsing rules           done! [3 rules in 1ms; 14 rules/s]
Problems detected       done! [0 in 1m24.687s; 0/s]
Reading my-gke-metrics  done! [28.01GB in 1m24.685s; 330.79MB/s]
Matching lines          done! [114.42M lines in 1m24.685s; 1.35M lines/s]

Wrote report to preq-report-1743656723.json

Command line options reference​

preq -h

Example output

Usage: preq [flags]

Flags:
  -h, --help              Show context-sensitive help.
  -a, --action=STRING     Path to an automated action or runbook config file
  -d, --disabled          Do not run community CREs
  -g, --generate          Generate data sources template
  -j, --cron              Generate Kubernetes cronjob template
  -l, --level=STRING      Print logs at this level to stderr
  -o, --name=STRING       Output name for reports, data source templates, or notifications
  -q, --quiet             Quiet mode, do not print progress
  -r, --rules=STRING      Path to a CRE rules file
  -s, --source=STRING     Path to a data source Yaml file
  -v, --version           Print version and exit
  -y, --accept-updates    Accept updates to rules or new release