Setting Log File and Configurations in DataHub CLI for Debugging and Rotation

ingestion
1

Original Slack Thread

<@U06TM7M40RK> cli에서 log파일 지정은 어떻게?

2

To specify a log file for the DataHub CLI, you can use the --log-file option. This option allows you to write all logs and stdout to the specified file, and it will contain debug logs regardless of the user’s CLI arguments. Here’s an example of how you can use it:

datahub --log-file /path/to/your/logfile.log

This command will direct the logs to the specified file path (/path/to/your/logfile.log).

Would you like more detail on any specific part of this configuration?

Sources:

3

<@U06TM7M40RK> log파일 레벨과 형식, 그리고, rotation방법은 어떻게 지정하나?

4

To configure the log file level, format, and rotation for DataHub, you typically need to modify the logback.xml configuration file. This file controls how logging is handled in DataHub, including the log level, format, and rotation policies.

Log Level and Format
You can set the log level and format by configuring the logback.xml file. Here is an example configuration:

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/path/to/your/logfile.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <!-- rollover daily and when the file reaches 10MB -->
      <fileNamePattern>/path/to/your/logfile.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
      <maxFileSize>10MB</maxFileSize>
      <maxHistory>30</maxHistory>
      <totalSizeCap>1GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss} - %msg%n</pattern>
    </encoder>
  </appender>
  <root level="info">
    <appender-ref ref="FILE" />
  </root>
</configuration>

Log Rotation
The SizeAndTimeBasedRollingPolicy is used for log rotation. In the example above:

  • <fileNamePattern> specifies the pattern for the log files, including date and index.
  • <maxFileSize> sets the maximum size of a log file before it is rolled over.
  • <maxHistory> determines how many rollover files to keep.
  • <totalSizeCap> limits the total size of all log files.

Applying the Configuration
To apply this configuration, ensure that your DataHub instance is using this logback.xml file. You may need to set the JAVA_OPTS environment variable to point to your custom logback.xml file:

export JAVA_OPTS="-Dlogback.configurationFile=/path/to/your/logback.xml"

Would you like more detailed steps on any of these configurations?

Sources:

5

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><