If you need to monitor your cluster, tune performance, or troubleshoot issues, you can check the CockroachDB logs, which include details about notable cluster, node, and range-level events.
Details
When a node starts, shuts down, or processes a cockroach
command, it produces a stream of messages about the command's activities. Each message is composed of:
- A payload that contains events either structured in JSON or conveyed in an arbitrary string. For details on structured event types and their fields, see Notable Event Types.
- An envelope that contains event metadata (e.g., severity, date, timestamp, channel). Depending on the log format you specify when configuring logs, the envelope can be formatted either in JSON or as a flat prefix to the message.
Messages are organized into appropriate logging channels and then routed through log sinks. Each sink further processes and filters the messages before emitting them to destinations outside CockroachDB. The mapping of channels to sinks, as well as the processing and filtering done by each sink, is configurable and is intended to support multiple use cases. For more information on configuring sinks and severity levels, see Configure Logs.
All cockroach
commands support logging. However, note that most messages related to cluster operation are generated by cockroach start
or cockroach start-single-node
. Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands.
Logging channels
Log channels cannot be completely disabled. You can configure how log messages are routed to different sinks and adjust verbosity levels, but you cannot turn off a log channel entirely. For example, a channel that is not explicitly configured for a file group is enabled in the default
file group.
Log messages in CockroachDB are directed to logging channels, which can in turn be assigned to output to one or more log sinks.
This allows you to group channels that log related information (e.g., operational, security, or SQL events) into their own sinks. Each sink can output to a predetermined destination where the logs can be collected and parsed. For usage examples, see Logging Use Cases.
Channel | Description |
---|---|
New in v25.4: CHANGEFEED |
Changefeed events. Note: Changefeed telemetry events currently sent to the TELEMETRY channel will be routed to this channel in a future release.To assess potential downstream impacts on your logging setup and pipelines, set the log.channel_compatibility_mode.enabled cluster setting to false in a non-production environment. This will log these events to this CHANGEFEED channel. |
DEV |
Uncategorized and debug messages. |
KV_DISTRIBUTION |
Data distribution events, such as moving replicas between stores in a cluster, adding replicas to ranges, and removing replicas from ranges. |
New in v25.4: KV_EXEC |
KV execution events that are not related to distribution. |
HEALTH |
Resource usage; node-node connection events, including connection errors; up- and down-replication and range unavailability. |
OPS |
Process starts, stops, shutdowns, and crashes (if they can be logged); changes to cluster topology, such as node additions, removals, and decommissions. |
PRIVILEGES |
Changes to privileges and object ownership. |
SENSITIVE_ACCESS |
SQL audit events (when enabled via ALTER TABLE ... EXPERIMENTAL_AUDIT ). |
SESSIONS |
Client connections and disconnections (when enabled via the server.auth_log.sql_connections.enabled cluster setting); SQL authentication logins/attempts and session/query terminations (when enabled via the server.auth_log.sql_sessions.enabled cluster setting). |
SQL_EXEC |
SQL statement executions (when enabled via the sql.trace.log_statement_execute cluster setting) and uncaught Go panic errors during SQL statement execution.Warning: Logging cluster-wide executions by enabling the sql.trace.log_statement_execute cluster setting will incur considerable overhead and may have a negative performance impact.Note: In a future release, the following events will be routed to this channel:To assess potential downstream impacts on your logging setup and pipelines, set the log.channel_compatibility_mode.enabled cluster setting to false in a non-production environment. This will log these events to this SQL_EXEC channel. |
SQL_PERF |
SQL executions that impact performance, such as slow queries (when enabled via the sql.log.slow_query.latency_threshold and/or sql.log.slow_query.experimental_full_table_scans.enabled cluster settings).Note: As of v25.4, the SQL_PERF and the SQL_INTERNAL_PERF channels are deprecated and will be removed in a future release. SQL Slow Query Log and SQL Slow Query Log (Internal) events currently sent to the SQL_PERF and SQL_INTERNAL_PERF channels, respectively, will be routed to the SQL_EXEC channel in a future release.To assess potential downstream impacts on your logging setup and pipelines, set the log.channel_compatibility_mode.enabled cluster setting to false in a non-production environment. This will log these events to the SQL_EXEC channel. |
SQL_SCHEMA |
Database, schema, table, sequence, view, and type creation; changes to table columns and sequence parameters. |
STORAGE |
Low-level storage logs from the Pebble storage engine. |
TELEMETRY |
Telemetry events for internal usage, except for sampled_query and sampled_transaction events which may be logged externally to Datadog.Note: sampled_query and sampled_transaction events currently sent to this channel will be routed to the SQL_EXEC channel in a future release. Changefeed telemetry events currently sent to this channel will be routed to the CHANGEFEED channel in a future release.To assess potential downstream impacts on your logging setup and pipelines, set the log.channel_compatibility_mode.enabled cluster setting to false in a non-production environment. This will log these events to the SQL_EXEC channel. |
USER_ADMIN |
Changes to users, roles, and authentication credentials. |
Logging channels are analogous to logging facilities in Syslog or logging services in Datadog. For more details on the contents of each logging channel, see the Logging reference.
log.channel_compatibility_mode.enabled
New in v25.4:
The log.channel_compatibility_mode.enabled
cluster setting when set to:
true
(default): Logs are sent to the expected logging channels. Specifically:- Changefeed telemetry events are sent to the
TELEMETRY
channel. - SQL Slow Query Log and SQL Slow Query Log (Internal) events are sent to the
SQL_PERF
andSQL_INTERNAL_PERF
channels, respectively. sampled_query
andsampled_transaction
events are sent to theTELEMETRY
channel.
- Changefeed telemetry events are sent to the
false
: Logs are sent to different logging channels as part of the logging channel consolidation effort. Specifically:- Changefeed telemetry events are sent to the
CHANGEFEED
channel. - SQL Slow Query Log and SQL Slow Query Log (Internal) events are sent to the
SQL_EXEC
channel. sampled_query
andsampled_transaction
events are sent to theSQL_EXEC
channel.
- Changefeed telemetry events are sent to the
In future releases, log.channel_compatibility_mode.enabled
will default to false
and then this cluster setting will be removed. To assess potential downstream impacts on your logging setup and pipelines, set the log.channel_compatibility_mode.enabled
cluster setting to false
in a non-production environment.
Logging destinations
When using the default logging configuration, the events collected on each logging channel are split into log files as follows:
Filename | Description | Channels |
---|---|---|
cockroach.log |
General CockroachDB log | DEV |
cockroach-changefeed.log |
Changefeed log | CHANGEFEED |
cockroach-health.log |
Health log | HEALTH |
cockroach-kv-distribution.log |
Data distribution log | KV_DISTRIBUTION |
cockroach-kv-exec.log |
KV execution log | KV_EXEC |
cockroach-security.log |
SQL security log | PRIVILEGES + USER_ADMIN |
cockroach-sql-audit.log |
SQL access audit log | SENSITIVE_ACCESS |
cockroach-sql-auth.log |
SQL authentication log | SESSIONS |
cockroach-sql-exec.log |
SQL execution log | SQL_EXEC |
cockroach-sql-slow.log |
SQL slow query log | SQL_PERF |
cockroach-sql-schema.log |
SQL schema change log | SQL_SCHEMA |
cockroach-pebble.log |
Pebble log | STORAGE |
cockroach-telemetry.log |
Telemetry log | TELEMETRY |
Logging destinations and many other parameters are configurable via YAML. For details, see Configure Logs.