LogLite: Lightweight Plug-and-Play Streaming Log Compression

Benzhao Tang; Shiyu Yang; Wenjie Zhang; Xuemin Lin; Zhihong Tian; Zhitao Shen

REVIEW 3 major objections 4 minor 62 references

LogLite: Lightweight Plug-and-Play Streaming Log Compression

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Log length plus XOR compresses logs up to 67.8% tighter, with no templates or training.

desk verdict LogLite is a genuinely useful streaming log compressor, but the lossless claim rests on an unstated NUL-free assumption and the characterization study moves the goalposts on the threshold. read the letter →

arxiv 2507.10337 v1 pith:5OMT3VZ4 submitted 2025-07-14 cs.DB

Benzhao Tang , Shiyu Yang , Zhitao Shen , Wenjie Zhang , Xuemin Lin , Zhihong Tian This is my paper · ORCID

classification cs.DB

keywords logcompressionstreaminglosslessXORencodingrun-lengthlengthJSONlogsParetooptimality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a log line's length is a reliable, training-free signal for finding a nearby line that is nearly identical at aligned character positions, and that this fact alone can power a streaming lossless compressor competitive with state-of-the-art log-specific and general-purpose methods. On 21 public log datasets, LogLite is reported to reach the Pareto frontier in seven of eight speed-versus-ratio settings, with average compression-ratio improvements up to 67.8% over the best baseline in line-by-line JSON compression and up to 2.7 times faster compression. The algorithm caches recent lines grouped by length, XORs a new line against a similar cached line to turn matching characters into null bytes, and run-length encodes the null runs. Because it needs no predefined rules, sampling, or training, it can compress each log immediately at generation time and adapt as log formats evolve.

What carries the argument

L-Windows is a hash table of FIFO queues, one per observed log length, holding at most k recent lines of that length; it supplies the candidate reference line. XOR-Preserve computes a character-wise XOR between the new line and the best candidate, keeps all null bytes, and copies the new line's original characters at differing positions so that decompression never needs an XOR operation. RLE-b/B then encodes runs of null bytes, either as a bit stream for speed or as a byte-aligned stream that can be fed to a general-purpose compressor; a small header records the window ID, similarity flag, and alignment.

What would settle it

Run LogLite with default settings on an artificial log stream generated from a template whose variable fields are padded to random lengths so that same-length lines share fewer than 85% of their positions; the compression ratio should approach or exceed 1.0 as the best match falls below threshold. A second direct test would measure the same-length match proportion on OpenStack and HDFS at theta = 0.85 and check whether the published ratio gap on those datasets tracks the drop in the match proportion.

Watch

Extended reading notes

Core claim

The central claim is that logs of the same length are overwhelmingly likely to be similar at aligned character positions, so log length can replace template parsing as the organizing principle for lossless log compression. LogLite operationalizes this by maintaining length-keyed FIFO windows of recent lines, searching from newest to oldest for a line whose XOR-based similarity exceeds a threshold, and encoding the result with run-length coding. The paper reports that this yields smaller output than existing log-specific and general-purpose baselines in most line-by-line scenarios, and that when the byte-stream variant is followed by Zstd or LZMA it remains competitive on archived files while running far faster than template-based log compressors.

Load-bearing premise

The whole method rests on the premise that a recently seen line of the same length will share most characters at the same positions, so that XORing them produces long runs of null bytes; the authors themselves note that OpenStack and HDFS fall short at the default 0.85 threshold and only recover the match proportion by lowering it to 0.75.

Editorial extensions

If this is right

A log producer can compress each line at the moment it is generated, so the bytes written to disk and sent over the network shrink before any batching, with memory bounded by the length-keyed windows.
No dictionary, template, or training phase means format drift from TEXT to JSON or new variable fields does not invalidate the compressor; it only creates new length buckets.
The byte-aligned variant composes with Zstd or LZMA, so archived log files can be made smaller than LZMA alone while compressing faster than template-based log-specific tools.
Decompression stays cheap because the preserved original characters at differences mean recovery is run-length decoding plus substitution from the cached reference line.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A consequence the authors leave implicit is that the mechanism never inspects field names or structure, so it should transfer to any line-oriented machine-generated text such as config dumps, CSV exports, or telemetry, provided line lengths stay finite and same-length lines stay aligned.
The similarity threshold theta is doing more work than the paper emphasizes: lowering it from 0.85 to 0.75 recovers the match proportion for OpenStack and HDFS, so an adaptive threshold that watches the recent match rate would make the compressor more robust on variable-heavy logs.
A testable extension would be self-tuning window size k or threshold theta per length bucket, since the characterization data show large variation across datasets; this could preserve the plug-and-play property while closing the gap on datasets like HDFS.
The XOR-plus-RLE scheme is a lightweight form of delta compression, so maintaining a small set of representative lines per length instead of only the most recent k lines could trade a little memory for higher hit rates and better ratios.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

LogLite earns a serious look. The core idea—group log lines by exact byte length, XOR with a recent same-length reference, preserve original characters in mismatching positions, then run-length encode the zeros—is simple and effective, and as far as I know it's new in this form. The paper backs it with a careful characterization study over 21 public datasets, a thorough comparison against both general-purpose and log-specific baselines in line-by-line and file modes, an ablation, a PostgreSQL case study, and an open-source repo. The numbers are credible: LogLite-b improves compression ratio over the best dictionary-based baseline by 20% on TEXT and 67% on JSON, while running faster than most competitors. The plug-and-play, streaming, no-training property is real and practically valuable.

The soft spots are real but addressable. The biggest is a correctness gap the paper doesn't acknowledge: the encoding uses '\0' as the match marker. If a log line contains a literal NUL byte at a position where the reference byte is non-NUL, XOR-P writes '\0' for that position, and the decoder mistakes it for a match and overwrites it with the reference byte. The lossless claim therefore assumes NUL-free input, which is never stated or tested. For TEXT and JSON logs this is rare, but it's a genuine hole in the main claim and should be fixed or caveated.

Second, the characterization study changes the similarity threshold from 0.85 to 0.75 for OpenStack and HDFS after seeing the results, to make Observation 2 look stronger. That's post hoc, and it undercuts the generality of the observation for those datasets. The paper should report both numbers transparently and explain why the adjustment is legitimate.

Third, all numbers come from single runs; no variance, no multiple trials. The Pareto-optimality claim is built on aggregates, and the default k=8 and theta=0.85 were presumably chosen using these same datasets. These are presentation/reproducibility concerns, not fatal flaws.

The central approach holds up. The NUL issue is small in practice but important to state; the threshold issue is a matter of presentation. This is a solid, useful contribution to log compression, and the open-source code makes the claims checkable. I'd send it to peer review without hesitation.

Referee Report

3 major / 4 minor

Summary. The paper presents LogLite, a streaming, training-free, lossless log compression method for TEXT and JSON logs. It first reports a characterization study of 16 Loghub TEXT datasets and 5 μSlope JSON datasets, claiming four observations about the finiteness of log lengths, the similarity of same-length logs, the advantage of reverse-order adjacency search, and the upper bound on similarity. The method uses length-indexed FIFO windows (L-Windows), a character-preserving XOR encoding (XOR-P), and byte- or bit-oriented run-length encoding (RLE-B/RLE-b), with optional post-compression by Zstd or LZMA. The evaluation compares LogLite against log-specific and general-purpose baselines for line-by-line and file compression, reports Pareto optimality in seven of eight aggregate scatter plots, and includes a PostgreSQL case study, an ablation study, and a parameter-sensitivity analysis.

Significance. If the correctness and evaluation issues are resolved, this is a useful contribution: it gives an open-sourced, plug-and-play, streaming compressor with a transparent and efficient design, and the length-based characterization is a reusable empirical result. The core XOR-P/RLE idea is simple and well motivated by the observation that aligned same-length logs share most characters. The claim that LogLite is Pareto-optimal in most scenarios is currently supported only by aggregate plots and default parameters chosen on the same datasets used for evaluation, so the quantitative headline should be treated with caution. The stress-test concern about a missing below-threshold fallback does not land: Section 3.4 explicitly selects the most similar cached log when no log exceeds θ, so low similarity degrades the compression ratio rather than correctness.

major comments (3)

[Section 3.4, Figure 4] The lossless claim is not valid for inputs containing NUL bytes. In XOR-P, a '\0' byte is the marker for a character match. When the current log's original byte at a mismatching position is itself '\0' and the reference byte is non-NUL, the XOR result at that position is non-NUL, so 'Preserving Original Characters' writes '\0' into the encoded stream; the decoder then treats that '\0' as a match and replaces it with the reference byte, losing the original NUL. The abstract and Section 4 claim lossless compression for TEXT and JSON logs without stating a NUL-free input assumption. Please either state and test the NUL-free assumption explicitly, or amend the encoding so that a literal NUL can be distinguished from a match marker.
[Section 2, Observation 2 and Section 4.1.3] The characterization reports that OpenStack and HDFS only exceed 98.83% PSL after lowering the similarity threshold from the default 0.85 to 0.75, yet the compression evaluation fixes θ=0.85 for all datasets. In addition, the default parameters (k=8, θ=0.85) appear to be selected using the same datasets that are later evaluated. The paper should report PSL and compression results for low-PSL datasets at the actual threshold used, and should tune parameters on a held-out subset or otherwise address the selection-on-evaluation-data concern. As written, the 'Pareto optimality in most scenarios' claim rests on parameters chosen from the test data.
[Section 4.2.3 and Tables 2–4] The Pareto-optimality claim is assessed only through eight aggregate scatter plots, with no per-dataset Pareto accounting and no variance information. No standard deviations, confidence intervals, or multiple-run results are reported for compression ratio or speed, so the reader cannot tell how many of the 21 datasets actually lie on the frontier or whether the 'seven out of eight figures' statement is stable. Please provide per-dataset Pareto counts and basic variance statistics for the headline numbers.

minor comments (4)

[Section 1, contribution list] The sentence 'we propose LogLite, which is, to the best of our knowledge, which is the first ...' contains a duplicated 'which is'; please rephrase.
[Section 4.5] The paragraph beginning 'In summary' appears twice in immediate succession; one copy should be deleted.
[Table 1] The dataset labeled 'Elasticserch' should be 'Elasticsearch'.
[Section 4.2.2] The text refers to 'LogLite-L' in the decompression-speed discussion, but the method is named 'LogLite-BL' in Table 4; please make the notation consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the observations are empirical, the method is benchmarked on independent public data, and no load-bearing step reduces to its own inputs.

full rationale

LogLite's derivation chain is not circular. The four observations in Section 2 are empirical measurements on public Loghub and μSlope datasets, with similarity, PSL, search counts, and MSS all defined before any compressor design. The method in Section 3 uses length bucketing, XOR-P, and RLE-b/B as design choices motivated by those observations; there is no equation in which a claimed prediction is inserted as an input or in which the method's output is defined as its target. The evaluation in Section 4 compares measured compression ratios, speeds, and throughput against independent baselines on the same public datasets, so the headline claims are externally benchmarked rather than derived from assumptions. The use of default parameters k=8 and theta=0.85, with parameter sensitivity studied on a subset of the evaluation datasets, is a parameter-selection and overfitting concern, not circular reasoning. The only self-citation, reference [55] for PBC, is used as a baseline and not as load-bearing justification for LogLite's design. The unstated NUL-free input assumption in XOR-P and RLE-b/B is a real correctness gap—a '\0' byte in a log at a position where the reference byte is non-'\0' can be misread as a match marker and corrupted on decompression—but that is a soundness issue, not circularity. Since no load-bearing step reduces by the paper's own equations or by self-citation to its own inputs, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces algorithmic components (L-Windows, XOR-P, RLE-b/B) rather than physical entities, so invented_entities is empty. The central claim depends on two tuned parameters and on the empirical premise that same-length logs align well, plus an unstated no-null-byte assumption.

free parameters (2)

window size k = 8 (default; 2-32 considered)
Cache size per length bucket; controls memory versus similarity opportunities; tuned in Section 4.5.
similarity threshold theta = 0.85 (default; 0.75 used post hoc for OpenStack/HDFS in Observation 2)
Controls whether a candidate is accepted as similar; affects compression ratio and speed; tuned in Section 4.5.

assumptions (4)

standard math XOR is reversible and RLE encodings can be decoded given length and instruction metadata
Foundation of XOR-P and RLE-b/B in Sections 3.4 and 3.5.
domain assumption Logs from the tested systems consist of same-length entries with high aligned similarity
Core enabling premise for L-Windows and XOR-P; Observation 2 and Table 1 report PSL, but OpenStack and HDFS need a lower threshold to reach high PSL.
domain assumption Input log entries contain no null ('\0') characters
XOR-P uses '\0' as the 'same as reference' marker; a literal null byte in input would be ambiguous and break losslessness. This is never stated in the paper.
domain assumption The 21 public datasets are representative of production log workloads
Evaluation and generality claims rest on Loghub and muSlope datasets as described in Section 4.1.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LogLite: Lightweight Plug-and-Play Streaming Log Compression." pith.science (2026). https://pith.science/paper/5OMT3VZ4

@misc{pith2026250710337,
  author       = {Pith},
  title        = {Pith review of: LogLite: Lightweight Plug-and-Play Streaming Log Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OMT3VZ4}},
  note         = {Machine review of arXiv:2507.10337}
}

abstract

Log data is a vital resource for capturing system events and states. With the increasing complexity and widespread adoption ofmodern software systems and IoT devices, the daily volume of log generation has surged to tens of petabytes, leading to significant collection and storage costs. To address this challenge, lossless log compression has emerged as an effective solution, enabling substantial resource savings without compromising log information. In this paper, we first conduct a characterization study on extensive public log datasets and identify four key observations. Building on these insights, we propose LogLite, a lightweight, plug-and-play, streaming lossless compression algorithm designed to handle both TEXT and JSON logs throughout their life cycle. LogLite requires no predefined rules or pre-training and is inherently adaptable to evolving log structures. Our evaluation shows that, compared to state-of-the-art baselines, LogLite achieves Pareto optimality in most scenarios, delivering an average improvement of up to 67.8% in compression ratio and up to 2.7 $\times$ in compression speed.

Figures

Figures reproduced from arXiv: 2507.10337 by the authors.

**Figure 1.** The life cycle of log data. ranging from hundreds of terabytes (TB) to tens of petabytes (PB) daily. For instance, Alibaba’s IoT devices transmit hundreds of TB of logs each day [4]; Uber generates over 10 PB of logs during a busy day across all its services [46]; and large real-time messaging applications like WeChat produce 16–20 PB of logs daily [53]. Due to the importance of logs and regulatory requirements, th… view at source ↗

**Figure 2.** Samples from Apache log. similarity score is defined in Equation (1): 𝑆𝑖𝑚(log𝑛 , log𝑚) = Ílen(log𝑛 ) 𝑖=1 1{𝑐𝑖=𝑐 ′ 𝑖 } len(log𝑛 ) (1) where len(log𝑛 ) = len(log𝑚), and 1{𝑐𝑖=𝑐 ′ 𝑖 } is an indicator function that equals 1 when 𝑐𝑖 is identical to 𝑐 ′ 𝑖 , and 0 otherwise. Adjacent Logs: The adjacent logs of log𝑛 are the subset (log𝑛−𝑘 , . . . , log𝑛−1 ), where 1 ≤ 𝑘 < 𝑛, representing the 𝑘 preceding logs. Additionally, l… view at source ↗

**Figure 3.** Overview of the LogLite framework (Logs of the same length have the same color). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Components of the LogLite framework. The design of XOR-based Similarity Search is based on the following two rationales: (1) Compared to other string similarity calculation methods, such as edit distance, XOR-based similarity score does not require expensive string m…

**Figure 5.** Figure 5: Performance visualization of compared baselines with Pareto front (red lines). [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Throughput of logs processing. The overall PostgreSQL log data compression ratios for LogLiteb, PBC, and Zstd-d are 0.1258, 0.5802, and 0.3255, respectively. Notably, LogLite-b is plug-and-play, whereas PBC and Zstd-d require offline sampling and training of their re…

**Figure 7.** Figure 7: Ablation study on 5 representative datasets. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Effect of different window size 𝑘 and similarity threshold 𝜃. further compression. While preserving original characters requires additional operations, the use of efficient SIMD implementations limits the average slowdown in compression speed to just 0.13%. Furthermore…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 57 canonical work pages

[1]

2025. LogLite. https://github.com/benzhaotang/LogLite

work page 2025
[2]

Vaibhav Agrawal, Devanjal Kotia, Kamelia Moshirian, and Mihui Kim. 2018. Log- based cloud monitoring system for OpenStack. In 2018 IEEE Fourth International Conference on Big Data Computing Service and Applications (BigDataService) . IEEE, 276–281

work page 2018
[3]

Jyrki Alakuijala, Andrea Farruggia, Paolo Ferragina, Eugene Kliuchnikov, Robert Obryk, Zoltan Szabadka, and Lode Vandevenne. 2018. Brotli: A general-purpose data compressor. ACM Transactions on Information Systems (TOIS) 37, 1 (2018), 1–30

work page 2018
[4]

aliyun. 2018. aliyunIoT. https://developer.aliyun.com/article/637406

work page 2018
[5]

Anunay Amar and Peter C Rigby. 2019. Mining historical test logs to predict bugs and localize faults in the test logs. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 140–151

work page 2019
[6]

Peter Boncz, Thomas Neumann, and Viktor Leis. 2020. FSST: fast random access string compression. Proceedings of the VLDB Endowment 13, 12 (2020), 2649–2661

2020
[7]

Wei Cao, Xiaojie Feng, Boyuan Liang, Tianyu Zhang, Yusong Gao, Yunyang Zhang, and Feifei Li. 2021. Logstore: A cloud-native and multi-tenant log database. In Proceedings of the 2021 International Conference on Management of Data . 2464– 2476

work page 2021
[8]

Robert Christensen and Feifei Li. 2013. Adaptive log compression for massive log data.. In SIGMOD Conference. 1283–1284

work page 2013

Show all 62 references

[9]

Yann Collet. 2011. LZ4: Fast Compression Algorithm. https://github.com/lz4/lz4

2011
[10]

Yann Collet and Murray Kucherawy. 2018. Zstandard Compression and the application/zstd Media Type. Technical Report

2018
[11]

Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security . 1285–1298

2017
[12]

Jarek Duda, Khalid Tahboub, Neeraj J Gadgil, and Edward J Delp. 2015. The use of asymmetric numeral systems as an accurate replacement for Huffman coding. In 2015 Picture Coding Symposium (PCS) . IEEE, 65–69

2015
[13]

Susan Dumais, Robin Jeffries, Daniel M Russell, Diane Tang, and Jaime Teevan
[14]

Yao-Chung Fan, Yu-Chi Chen, Kuan-Chieh Tung, Kuo-Chen Wu, and Arbee LP Chen. 2015. A framework for enabling user preference profiling through wi-fi logs. IEEE Transactions on Knowledge and Data Engineering 28, 3 (2015), 592–603

2015
[15]

Bettina Fazzinga, Sergio Flesca, Filippo Furfaro, and Luigi Pontieri. 2018. Online and offline classification of traces of event logs on the basis of security risks. Journal of Intelligent Information Systems 50 (2018), 195–230

2018
[16]

Bo Feng, Chentao Wu, and Jie Li. 2016. MLC: an efficient multi-level log compres- sion method for cloud backup systems. In 2016 IEEE Trustcom/BigDataSE/ISPA. IEEE, 1358–1365

2016
[17]

Jean-loup Gailly and Mark Adler. 1992. GNU gzip.GNU Operating System (1992)

1992
[18]

Shilin He, Qingwei Lin, Jian-Guang Lou, Hongyu Zhang, Michael R Lyu, and Dongmei Zhang. 2018. Identifying impactful service system problems via log analysis. In Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foun...

2018
[19]

Shengsheng Huang, Jie Huang, Jinquan Dai, Tao Xie, and Bo Huang. 2010. The HiBench benchmark suite: Characterization of the MapReduce-based data anal- ysis. In 2010 IEEE 26th International conference on data engineering workshops (ICDEW 2010). IEEE, 41–51

2010
[20]

David A. Huffman. 1952. A Method for the Construction of Minimum- Redundancy Codes. Proceedings of the IRE 40, 9 (1952), 1098–1101

1952
[21]

Facebook Inc. 2012. RocksDB: A Persistent Key-Value Store for Flash and RAM Storage. https://github.com/facebook/rocksdb

2012
[22]

Google Inc. 2011. Snappy: A Fast Compressor. https://github.com/google/snappy

2011
[23]

Insanity Industries. 2021. Pareto-optimal compression. https://insanity. industries/post/pareto-optimal-compression/

2021
[24]

Zhouyang Jia, Shanshan Li, Xiaodong Liu, Xiangke Liao, and Yunhuai Liu. 2018. SMARTLOG: Place error log statement by deep understanding of log intention. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 61–71

2018
[25]

Phil Katz. 1989. ZIP: A File Compression Standard. https://www.info-zip.org/

1989
[26]

Abraham Lempel and Jacob Ziv. 2001. LZMA Algorithm. http://www.7-zip.org/

2001
[27]

Roman Leshchinskiy. 2018. LZBench: Compression Benchmarking Tool. https: //github.com/lemire/LZBench

2018
[28]

Ruiyuan Li, Zheng Li, Yi Wu, Chao Chen, and Yu Zheng. 2023. Elf: Erasing-based lossless floating-point compression. Proceedings of the VLDB Endowment 16, 7 (2023), 1763–1776

2023
[29]

Xiaoyun Li, Pengfei Chen, Linxiao Jing, Zilong He, and Guangba Yu. 2022. Swiss- log: Robust anomaly detection and localization for interleaved unstructured logs. IEEE Transactions on Dependable and Secure Computing 20, 4 (2022), 2762–2780

2022
[30]

Xiaoyun Li, Hongyu Zhang, Van-Hoang Le, and Pengfei Chen. 2024. Logshrink: Effective log compression by leveraging commonality and variability of log data. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12

2024
[31]

Panagiotis Liakos, Katia Papakonstantinopoulou, and Yannis Kotidis. 2022. Chimp: efficient lossless floating point compression for time series databases. Proceedings of the VLDB Endowment 15, 11 (2022), 3058–3070

2022
[32]

Hao Lin, Jingyu Zhou, Bin Yao, Minyi Guo, and Jie Li. 2015. Cowic: A column- wise independent compression for log stream analysis. In 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing . IEEE, 21–30

2015
[33]

Jimmy Lin et al . 2010. YCSB: A Workload Generation Framework for Cloud Databases. https://github.com/brianfrankcooper/YCSB

2010
[34]

Chunwei Liu, John Paparrizos, and Aaron J Elmore. 2024. Adaedge: A dynamic compression selection framework for resource constrained devices. In 2024 IEEE 40th International Conference on Data Engineering (ICDE) . IEEE, 1506–1519

2024
[35]

Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, and Michael R Lyu. 2019. Logzip: Extracting hidden structures via iterative clustering for log compression. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 863–873

2019
[36]

Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using deep learning to generate complete log statements. In Proceedings of the 44th Interna- tional Conference on Software Engineering . 2279–2290

2022
[37]

Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al. 2019. Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs.. InIJCAI, Vol. 19. 4739–4745

2019
[38]

Alina Oprea, Zhou Li, Ting-Fang Yen, Sang H Chin, and Sumayah Alrwais. 2015. Detection of early-stage enterprise infection by mining large-scale log data. In 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. IEEE, 45–56

2015
[39]

R. Pasco. 1977. Source coding algorithms for fast data compression (Ph.D. Thesis abstr.). IEEE Transactions on Information Theory 23, 4 (1977), 548–548

1977
[40]

Tuomas Pelkonen, Scott Franklin, Justin Teller, Paul Cavallaro, Qi Huang, Justin Meza, and Kaushik Veeraraghavan. 2015. Gorilla: A fast, scalable, in-memory time series database. Proceedings of the VLDB Endowment 8, 12 (2015), 1816–1827

2015
[41]

Kirk Rodrigues, Yu Luo, and Ding Yuan. 2021. CLP: Efficient and scalable search on compressed text logs. In15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) . 183–198

2021
[42]

Guoping Rong, Shenghui Gu, Haifeng Shen, He Zhang, and Hongyu Kuang
[43]

Carl Martin Rosenberg and Leon Moonen. 2020. Spectrum-based log diagnosis. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . 1–12

2020
[44]

Amazon Web Services. 2012. Amazon Redshift: Data Warehousing Service. https://aws.amazon.com/redshift/

2012
[45]

Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. 2010. The hadoop distributed file system. In 2010 IEEE 26th symposium on mass storage systems and technologies (MSST) . Ieee, 1–10

2010
[46]

Rui Wang, Devin Gibson, Kirk Rodrigues, Yu Luo, Yun Zhang, Kaibo Wang, Yupeng Fu, Ting Chen, and Ding Yuan. 2024. 𝜇Slope: High Compression and Fast Search on Semi-Structured Logs. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . 529–544

2024
[47]

Zhiyi Wang and Shimin Chen. 2017. Exploiting common patterns for tree- structured data. In Proceedings of the 2017 ACM International Conference on Management of Data. 883–896

2017
[48]

Junyu Wei, Guangyan Zhang, Junchao Chen, Yang Wang, Weimin Zheng, Tingtao Sun, Jiesheng Wu, and Jiangwei Jiang. 2023. Loggrep: Fast and cheap cloud log storage by exploiting both static and runtime patterns. In Proceedings of the Eighteenth European Conference on Computer Syst...

2023
[49]

Junyu Wei, Guangyan Zhang, Yang Wang, Zhiwei Liu, Zhanyang Zhu, Junchao Chen, Tingtao Sun, and Qi Zhou. 2021. On the feasibility of parser-based log compression in Large-Scale cloud systems. In 19th USENIX Conference on File and Storage Technologies (FAST 21). 249–262

2021
[50]

Wikipedia. 2024. Zstandard. https://en.wikipedia.org/wiki/Zstd

2024
[51]

Wikipedia contributors. 2023. Deflate — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Deflate&oldid=1148886022

2023
[52]

de Pádua, Weiyi Shang, Steve Sporea, Andrei Toma, and Sarah Sajedi

Kundi Yao, Guilherme B. de Pádua, Weiyi Shang, Steve Sporea, Andrei Toma, and Sarah Sajedi. 2018. Log4perf: Suggesting logging locations for web-based systems’ performance monitoring. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering . 127–138

2018
[53]

Guangba Yu, Pengfei Chen, Pairui Li, Tianjun Weng, Haibing Zheng, Yuetang Deng, and Zibin Zheng. 2023. Logreducer: Identify and reduce log hotspots in kernel on the fly. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1763–1775

2023
[54]

Chenxi Zhang, Xin Peng, Chaofeng Sha, Ke Zhang, Zhenqing Fu, Xiya Wu, Qingwei Lin, and Dongmei Zhang. 2022. Deeptralog: Trace-log combined mi- croservice anomaly detection through graph-based deep learning. In Proceedings of the 44th international conference on software engine...

2022
[55]

Jiujing Zhang, Zhitao Shen, Shiyu Yang, Lingkai Meng, Chuan Xiao, Wei Jia, Yue Li, Qinhui Sun, Wenjie Zhang, and Xuemin Lin. 2023. High-Ratio Compression for Machine-Generated Data. Proceedings of the ACM on Management of Data 1, 4 (2023), 1–27

2023
[56]

Bolong Zheng, Yongyong Gao, Jingyi Wan, Lingsen Yan, Long Hu, Bo Liu, Yunjun Gao, Xiaofang Zhou, and Christian S. Jensen. 2023. DecLog: Decentralized Logging in Non-Volatile Memory for Time Series Database Systems. Proc. VLDB Endow. 17, 1 (2023), 1–14

2023
[57]

Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Dewei Liu, Qilin Xiang, and Chuan He. 2019. Latent error prediction and fault localization for microservice applications by learning from system trace logs. In Proceedings of the 2019 27th ACM joint meeting on European software ...

2019
[58]

Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R. Lyu. 2023. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. In IEEE International Symposium on Software Reliability Engineering (ISSRE)

2023
[59]

Jacob Ziv and Abraham Lempel. 1977. A universal algorithm for sequential data compression. IEEE Transactions on information theory 23, 3 (1977), 337–343

1977
[60]

Jacob Ziv and Abraham Lempel. 1978. Compression of individual sequences via variable-rate coding. IEEE transactions on Information Theory 24, 5 (1978), 530–536

1978
[2014]

Ways of Knowing in HCI (2014), 349–372

Understanding user behavior through log data and analysis. Ways of Knowing in HCI (2014), 349–372

2014
[2023]

In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

How Do Developers’ Profiles and Experiences Influence their Logging Practices? An Empirical Study of Industrial Practitioners. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 855–867

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

2025. LogLite. https://github.com/benzhaotang/LogLite

work page 2025

[2] [2]

Vaibhav Agrawal, Devanjal Kotia, Kamelia Moshirian, and Mihui Kim. 2018. Log- based cloud monitoring system for OpenStack. In 2018 IEEE Fourth International Conference on Big Data Computing Service and Applications (BigDataService) . IEEE, 276–281

work page 2018

[3] [3]

Jyrki Alakuijala, Andrea Farruggia, Paolo Ferragina, Eugene Kliuchnikov, Robert Obryk, Zoltan Szabadka, and Lode Vandevenne. 2018. Brotli: A general-purpose data compressor. ACM Transactions on Information Systems (TOIS) 37, 1 (2018), 1–30

work page 2018

[4] [4]

aliyun. 2018. aliyunIoT. https://developer.aliyun.com/article/637406

work page 2018

[5] [5]

Anunay Amar and Peter C Rigby. 2019. Mining historical test logs to predict bugs and localize faults in the test logs. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 140–151

work page 2019

[6] [6]

Peter Boncz, Thomas Neumann, and Viktor Leis. 2020. FSST: fast random access string compression. Proceedings of the VLDB Endowment 13, 12 (2020), 2649–2661

2020

[7] [7]

Wei Cao, Xiaojie Feng, Boyuan Liang, Tianyu Zhang, Yusong Gao, Yunyang Zhang, and Feifei Li. 2021. Logstore: A cloud-native and multi-tenant log database. In Proceedings of the 2021 International Conference on Management of Data . 2464– 2476

work page 2021

[8] [8]

Robert Christensen and Feifei Li. 2013. Adaptive log compression for massive log data.. In SIGMOD Conference. 1283–1284

work page 2013