Setting Up Fluentd Unified Logging

In Composer, you can use Fluentd as a logging layer to which you can direct the logs for various components of Composer. This enables you to customize the log output to meet the needs of your environment.

Composer leverages Fluentd’s unified logging layer to collect logs via a central API. Fluentd can be configured to aggregate logs to various data sources or outputs. For example, if you are directing all log files from your zoomdata-websocket.log and zoomdata-errors.log to a Fluentd server, you can add one of Fluentd’s plug-ins to write the log files to Elasticsearch to analyze web client errors for your environment.

Unified logging does not replace Composer's default logging architecture. It only augments that experience with an option for those wanting additional logging control.

By default, unified logging is disabled and needs to be enabled for each component you want captured in Fluentd.

Before You Begin

Composer’s unified logging configuration relies on Fluentd to write logs from Fluentd to a particular database or source. Consequently, Fluentd should be configured appropriately. For a list of compatible Fluentd plug-ins, refer to Fluentd's website and documentation. If written to a data source compatible with Composer’s connectors, Composer can then be used to analyze and visualize your logs.

Prerequisites

• A server with Fluentd installed and configured. For information and steps, refer to Fluentd's documentation.
• If Fluentd is not on the same server as Composer, you need the host and port numbers for the server.

Optional

• To output logs to another data source, that data source should be available and configured in conjunction with Fluentd to receive logs.

• To visualize logs within Composer, the data source referenced above should also be supported by Composer. For more information, see the Composer Data Connector Reference.

Steps

At a high level, the steps to set up Fluentd are as follows:

• Decide what microservices (such as zoomdata, query-engine, etc.) you want to log to Fluentd
• Enable unified logging for that particular microservice.
• Configure the host and port information for your Fluentd server to the properties file.

To set up unified logging for your Composer server:

1. On your server, access the .properties file of the microservice you want to run unified logging on. For example, etc/zoomdata/query-engine.properties. You can enable unified logging for these microservices:

   • Web microservices (zoomdata.properties)
   • Query engine (query-engine.properties)
   • Data source connectors (see Connector Properties and Property Files for a list of property files)
   • Data Writer framework (data-writer-postgresql.properties)

2. Define the following parameters the appropriate properties file:

   Property Syntax	Description
   logging.unified.level = <log-level>	Specify one of the following log levels for <log-level>: ERROR, WARN, INFO, DEBUG, TRACE, or OFF. If set to OFF, Fluentd unified logging is disabled.
   logging.unified.tag = <service-tag>	Specify the unique tag for the Composer microservice, used in grouping logs. This tag must be unique throughout Composer. Supply the Composer microservice name for <service-tag>. For example, logging.unified.tag = query-engine. This is important because the tag identifies which microservice the log message applies to. Valid values are query-engine, zoomdata-server, data-writer-postgresql, and edc-<connector-name> (where <connector-name> is one of the names listed in Composer Microservice Name Reference).
   logging.unified.label = <service-label>	Specify a more verbose label for the Composer microservice.
   logging.unified.host = <yourFluentdServerIPaddress>	Specify the IP address of the Fluentd logging server.
   logging.unified.port = <yourFluentdServerPort>	Specify the port of the Fluentd logging server.

3. Save your changes and exit the properties file.

4. Restart the Composer server.

On the Fluentd server, you can then direct your logs to a data source of your choice. For information and steps, see Fluentd's documentation on Output.