Data Sources

To run preq on data without using stdin, you can create a data sources configuration file.

Overview

A CRE rule consists of one or more abstract data sources used by preq, Prequel's open source problem detector, to target an addressable named data type (e.g. log, metric, trace, or event).

Because the location of data sources vary based on the environment (e.g. Kubernetes vs. virtual machines) or distribution (e.g. Docker vs. Bitnami), CRE rules are created to target a data source abstraction that is translated by the problem detector to the actual source's location in the environment.

The preq problem detector supports targeting log files via stdin (all abstract data sources) or by or addressed from the file system. Refer to Prequel for addressing containers or additional data types.

Data source configuration

When preq is run frequently on a system on multiple data sources, it's helpful to use a data source configuration to describe the locations of the abstract sources in relevant CRE rules.

example configuration

version: 0.0.1
sources:
  - name: myhadoop
    type: hadoop-hdfs
    desc: "HDFSv2"
    locations:
      - timestamp:
          regex: "^(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}) "
          format: "2006-01-02 15:04:05,999"
        path: /server/node_logs/hadoop-hdfs-datanode-mesos-*.log
  - name: my-gke-metrics
    type: gke-metrics-agent
    desc: "gke metrics-agent"
    locations:
      - timestamp:
          regex: "^\uFEFF?(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}), "
          format: "2006-01-02 15:04:05"
        path: /home/tony/Downloads/logs/Windows.log
  - name: mysyslog
    type: syslog
    desc: "Sample syslog"
    locations:
      - path: /var/log/syslog

Field	Description
name	Used to identify the data source for the user of the problem detector. This name is not used by CRE rules.
type	The standard abstract name for the data source. It is named in relevant CREs and used by problem detectors to resolve the data in the environment.
desc	Used to provide users with additional context on the data source.
locations	An ordered map of file system paths for the data source in the environment. Optionally a timestamp regex and format may be specified to describe the timestamp format in the data. Data will be searched in the order listed in the data source configuration.

Example

preq -s sources.yaml -r rules.yaml -d