Data Sources

To run preq on data without using stdin, you can create a data sources template file to describe where each relevant data source resides on your system. You can then provide this data source template to preq and it will automatically evaluate only the relevant CREs on your defined data sources.

tip

Clone https://github.com/prequel-dev/preq to follow along, copy/paste, and try the examples referenced below in the repo to learn how to use data source templates.

Generate a data source template

Use -g to generate a data source template from your installed CRE rules package.

Example:

$ preq -g

Example output

Wrote data source template to data-sources-0.3.12.yaml

Use -o - to print the template to standard out.

Example:

preq -g -o -

Example output (truncated):

# See https://docs.prequel.dev/data-sources for how to customize this template with your own data sources
# Remove any data sources that are not running on this system
# Add custom timestamp formats to the data sources if they are not already supported by default (see https://docs.prequel.dev/timestamps)
# If the data source is for a library that is used by multiple applications, you can add more than one path to the same data source
version: 0.3.12
sources:
- type: cre.log.neutron
  name: my-cre.log.neutron
  locations:
  - path: /path/to/my-cre.log.neutron
- type: cre.log.redis-py
  name: my-cre.log.redis-py
  locations:
  - path: /path/to/my-cre.log.redis-py
- type: cre.log.keda-operator.2.*
  name: my-cre.log.keda-operator.2.*
  locations:
  - path: /path/to/my-cre.log.keda-operator.2.*

Edit the template to point the data sources to the locations of the logs on your system. If the data source has a timestamp format that is not supported by default (see Default supported timestamps), then you can add them here.

add custom timestamps

version: 0.3.12
sources:
- type: cre.log.neutron
  name: my-cre.log.neutron
  locations:
  - path: /path/to/my-cre.log.neutron
    timestamp:
      regex: "^(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}) "
      format: "2006-01-02 15:04:05,999"

Use a data source template

Use -s to provide a data sources configuration file.

preq -s ./examples/40-sources.yaml 

Example output

Parsing rules           done! [3 rules in 1ms; 14 rules/s]
Problems detected       done! [0 in 1m24.687s; 0/s]
Reading my-gke-metrics  done! [28.01GB in 1m24.685s; 330.79MB/s]
Matching lines          done! [114.42M lines in 1m24.685s; 1.35M lines/s]

Wrote report to preq-report-1743656723.json

What are data sources?

A CRE rule consists of one or more abstract data sources used by preq to target log data.

Because the location of data sources vary based on the environment (e.g. Kubernetes vs. virtual machines) or distribution (e.g. Docker vs. Bitnami), CRE rules are created to target a data source abstraction that is translated by the problem detector to the actual source's location in the environment.

note

The preq problem detector supports targeting log files via stdin (all abstract data sources) or via logs on file system. Refer to Prequel for matching other types of data in distributed systems.

Data source configuration

When preq is run frequently on a system on multiple data sources, it's helpful to use a data source configuration to describe the locations of the abstract sources in relevant CRE rules.

examples/40-sources.yaml

version: 0.0.1
sources:
  - name: example-1
    type: cre.log.rabbitmq
    desc: "Example 1"
    locations:
      - path: ./examples/26-example.log
  - name: example-2
    type: cre.log.django
    desc: "Example 2"
    locations:
      - path: ./examples/25-example.log
  - name: example-3
    type: cre.log.nginx
    desc: "Example 3"
    locations:
      - path: ./examples/28-example.log
  - name: example-4
    type: cre.log.kafka
    desc: "Example 4"
    locations:
      - path: ./examples/23-example.log
      - path: ./examples/29-example.log

Field	Description
name	Used to identify the data source for the user of the problem detector. This name is not used by CRE rules.
type	The standard abstract name for the data source. It is named in relevant CREs and used by problem detectors to resolve the data in the environment.
desc	Used to provide users with additional context on the data source.
locations	An ordered map of file system paths for the data source in the environment. Optionally a timestamp regex and format may be specified to describe the timestamp format in the data. Data will be searched in the order listed in the data source configuration.

Example

preq -s sources.yaml -r rules.yaml -d

Scheduled jobs

Use crontab -e to run preq regularly with a data sources template.

crontab

*/15 * * * *  /path/to/bin/preq -s /path/to/data-sources.yaml -n /path/to/cre/reports/cre-report-`date +\%Y\%m\%d\%H\%M\%S`.json