Pith. sign in

REVIEW 42 references

Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A JEPA encoder finetuned on synthetic bar charts enables a lightweight decoder to recover bar values from chart images, but the method remains behind state-of-the-art supervised systems.

arxiv 2608.06062 v1 pith:WNLSNFKO submitted 2026-08-06 cs.CV

Poonam Poonam , Alexander Epple , Timo Ropinski This is my paper
classification cs.CV
keywords datafeaturesusedarchitecturebar-jepachartchartsconsuming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bar-JEPA is a pipeline for reading the numbers encoded in bar chart images. The system first takes a vision model trained with a self-supervised method called I-JEPA, which learns to predict missing patches of an image in a hidden representation space instead of reconstructing pixels. The authors finetune this model on 100,000 synthetic bar charts, then freeze it. A small decoder network is trained on the frozen features to output heatmaps showing the chart's origin, tick marks, and bar corners. At test time, a text recognition tool reads the numbers next to the tick marks, those numbers are matched to the detected tick positions, and a straight line is fit through them. The line converts each detected bar's pixel position into a numeric value.

The main experimental result is that finetuning the encoder on charts matters. Using the original ImageNet-pretrained encoder with no chart finetuning, the pipeline recovers almost no values on real chart images. After finetuning on synthetic charts, relaxed accuracy reaches about 45 percent on real data, and about 78 percent on synthetic test charts. Adding a second finetuning round on real-world charts raises real-world accuracy to about 50 percent. A variable-resolution input scheme, taken from Pix2Struct, adds a few points of accuracy.

The authors do not claim to beat state-of-the-art chart readers. They position the work as a demonstration that self-supervised feature learning can reduce the need for labeled chart data. The comparisons to prior supervised systems are made on different test sets, and the real-world test set appears to include chart types, such as stacked bars, that the training data explicitly excluded.

Extended reading notes

Core claim

The load-bearing assertion is: 'self-supervised finetuning on charts clearly is able to yield a significantly more capable encoder for downstream tasks' (Section 4.2). If true, the large gains over the vanilla ImageNet-1K encoder, from 0% to 42-50% relaxed value accuracy on real-world charts, are caused by chart-specific JEPA finetuning rather than by decoder training.

Load-bearing premise

The real-world evaluation is treated as in-scope for a model whose synthetic training explicitly excludes stacked bars, error bars, and 3D effects. The paper itself states that 'most of the charts in our test set do not fulfill the criteria Zhou et al. assume (e.g. include error bars / stacked bars)' (Section 4.3), so the reported real-world accuracy is likely measured partly on chart types the method was never designed to handle.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The pipeline is empirical and rests on domain assumptions about charts rather than on a derivation. The hand-set loss weights and detection thresholds affect results, and the evaluation assumes the real-world test charts are within the model's design scope despite evidence to the contrary.

free parameters (4)
  • Heatmap loss weights = lambda1=1.0, lambda2=1.0, lambda3=4.0, lambda4=4.0
    Hand-selected weights in Equation 5 balance origin, classification, regression, and keypoint losses; they affect the quality of detected bar and tick positions.
  • Detection confidence threshold = 0.75
    Candidates with confidence below 0.75 are discarded in Section 3.4; this directly controls precision and recall of bar and tick detections.
  • NMS radius factor = 1.5
    The non-maximum suppression radius in Equation 6 is set to 1.5 pixels in heatmap space; it determines how close two detections can be before one is suppressed.
  • Tick-label matching distance factor = 5.0
    A tick label is matched to a tick position only if the distance is at most 5.0 times the NMS radius; this threshold controls which OCR detections are used for value recovery.
assumptions (4)
  • domain assumption Value recovery assumes bar values are an affine function of pixel position, fitted by RANSAC over tick positions and labels.
    Used in Section 3.4; breaks for logarithmic, broken, or otherwise nonlinear value axes.
  • domain assumption Self-supervised finetuning on 100k synthetic charts transfers to real-world bar charts.
    The encoder is finetuned only on synthetic data; the method's real-world success depends on the synthetic distribution matching real chart appearance closely enough.
  • domain assumption The UB PMC test charts are inside the method's design scope, meaning simple vertical bars without stacking or error bars.
    Section 4.3 states that most of the real-world test charts contain stacked bars or error bars, which contradicts the training assumptions in Section 3.1.
  • domain assumption OCR text detections can be correctly matched to tick positions using the Hungarian algorithm with a distance threshold.
    Section 3.4 relies on legible, correctly placed tick labels; rotated, overlapping, or missing labels would break the matching step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture." pith.science (2026). https://pith.science/paper/WNLSNFKO

@misc{pith2026260806062,
  author       = {Pith},
  title        = {Pith review of: Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNLSNFKO}},
  note         = {Machine review of arXiv:2608.06062}
}
read the original abstract

Bar charts are commonly used in data visualization, and while they are easily understood by humans, it is non-trivial to extract the underlying data computationally. For a machine-learning-based approach, training chart de-rendering models usually requires labeled, real-world data. Labeling data is a time consuming task, which is why annotated data is scarce. Models can learn more efficiently when provided with features of high semantic quality, which a joint-embedding predictive architecture (JEPA) is designed to learn in a self-supervised manner. We present a per-bar, numerical value recovery pipeline for bar charts, where a JEPA encoder is used to produce semantically rich latent features. The decoder model consuming these features is simple and quick to train and outputs the coordinates of ticks and bars, which can be used to recover bar values. The effectiveness of self-supervised finetuning and quality of the extracted features is evident when comparing our model to end-to-end supervised baselines. Code, datasets and checkpoints are available on \href{https://github.com/dralois/Bar-JEPA}{GitHub}.

Figures

Figures reproduced from arXiv: 2608.06062 by the authors.

Figure 1
Figure 1. Examples of generated bar charts as used for encoder finetuning and de￾coder training. Charts are generated ac￾cording to the parameters in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Aspect-ratio preserving patch extraction ensures at most N patches are ob￾tained after resizing an image. The operation guarantees the new size to be close to the original aspect-ratio as much as possible. The chart is an example from [9]. Targets. For each image, target blocks are obtained by first sampling i blocks Bi from the M input patches. The size and aspect-ratio of each Bi is random and the blocks can overl… view at source ↗
Figure 4
Figure 4. The decoder pipeline consists of the frozen, pretrained I-JEPA encoder (a) and a decoder (b), which produces key point heatmaps. The heatmaps are consumed by the output heads (e), which in turn generate a combined heatmap. There are two possible configurations, a simple (d) and a slightly more sophisticated (c) decoder model. The chart is an example from [9]. each consisting of a deconvolution, batch normalization a… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An example of the heatmaps produced by the decoder. From left to right: Input image, (inverted) background map, top right bar corners, ticks and input image overlaid with all three heatmaps. Ltotal = λ1Lorg + λ2Lcls + λ3Lreg + λ4Lkp (5) Lorg: The origin loss is compute…
Figure 6
Figure 6. Figure 6: The confusion matrix of the simple decoder evaluated on [9], showing that most bars and ticks are not detected [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Simple vs. classic decoder on a chart from [9]. The activations of the simple decoder are blurry, whereas the classic decoder produces pronounced, sharp peaks [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 32 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF CVPR (2023)

    Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., Le- Cun, Y., Ballas, N.: Self-supervised learning from images with a joint-embedding predictive architecture. In: Proceedings of the IEEE/CVF CVPR (2023)

    work page 2023

  2. [2]

    arXiv preprint arXiv:2506.09985 (2025)

    Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al.: V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985 (2025)

    arXiv 2025

  3. [3]

    Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: V-JEPA: Latent video prediction for visual representation learning (2024), https://openreview.net/forum?id=WFYbBOEOtv, [Accessed 13-02-2026]

    work page 2024

  4. [4]

    In: ECCV

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: ECCV. Springer (2020) Bar-JEPA 15

    work page 2020

  5. [5]

    In- ternational Journal of Computer Vision132(1) (2024)

    Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., Wang, J.: Context autoencoder for self-supervised representation learning. In- ternational Journal of Computer Vision132(1) (2024)

    work page 2024

  6. [6]

    In: Proceedings of the IEEE/CVF ICCV (2023)

    Cheng, Z.Q., Dai, Q., Hauptmann, A.G.: Chartreader: A unified framework for chart derendering and comprehension without heuristic rules. In: Proceedings of the IEEE/CVF ICCV (2023)

    work page 2023

  7. [7]

    arXiv preprint arXiv:2507.05595 (2025)

    Cui, C., Sun, T., Lin, M., Gao, T., Zhang, Y., Liu, J., Wang, X., Zhang, Z., Zhou, C., Liu, H., et al.: Paddleocr 3.0 technical report. arXiv preprint arXiv:2507.05595 (2025)

    arXiv 2025

  8. [8]

    In: ICPR

    Davila, K., Lazarus, R., Xu, F., Rodríguez Alcántara, N., Setlur, S., Govindaraju, V., Mondal, A., Jawahar, C.: Chart-info 2024: A dataset for chart analysis and recognition. In: ICPR. Springer (2024)

    work page 2024

Show all 42 references
  1. [9]

    In: 2022 26th ICPR

    Davila, K., Xu, F., Ahmed, S., Mendoza, D.A., Setlur, S., Govindaraju, V.: Icpr 2022: Challenge on harvesting raw tables from infographics (chart-infographics). In: 2022 26th ICPR. IEEE (2022)

    2022

  2. [10]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recogni- tion at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  3. [11]

    arXiv preprint arXiv:2311.15830 (2023)

    Fei, Z., Fan, M., Huang, J.: A-jepa: Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830 (2023)

    2023 arXiv

  4. [12]

    Communi- cations of the ACM24(6) (1981)

    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communi- cations of the ACM24(6) (1981)

    1981

  5. [13]

    In: Proceedings of the IEEE/CVF WACV (2023)

    Hassan, M.Y., Singh, M., et al.: Lineex: Data extraction from scientific line charts. In: Proceedings of the IEEE/CVF WACV (2023)

    2023

  6. [14]

    In: Proceedings of the IEEE/CVF CVPR (2022)

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF CVPR (2022)

    2022

  7. [15]

    In: Proceedings of the 2007 ACM symposium on Document engineering (2007)

    Huang, W., Tan, C.L.: A system for understanding imaged infographics and its ap- plications. In: Proceedings of the 2007 ACM symposium on Document engineering (2007)

    2007

  8. [16]

    Computing in Science & Engineering9(3) (2007)

    Hunter, J.D.: Matplotlib: A 2d graphics environment. Computing in Science & Engineering9(3) (2007). https://doi.org/10.1109/MCSE.2007.55, [Accessed 07-06- 2026]

    2007 doi

  9. [17]

    arXiv preprint arXiv:2109.12442 (2021)

    Joshi, I.A.: Unblind the charts: Towards making interactive charts accessible in android applications. arXiv preprint arXiv:2109.12442 (2021)

    2021 arXiv

  10. [18]

    In: Proceedings of the 60th Annual Meeting of the ACL (Volume 1: Long Papers) (2022)

    Kantharaj, S., Leong, R.T., Lin, X., Masry, A., Thakkar, M., Hoque, E., Joty, S.: Chart-to-text: A large-scale benchmark for chart summarization. In: Proceedings of the 60th Annual Meeting of the ACL (Volume 1: Long Papers) (2022)

    2022

  11. [19]

    Naval research logistics quarterly2(1-2) (1955)

    Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2) (1955)

    1955

  12. [20]

    2, 2022- 06-27

    LeCun, Y.: A path towards autonomous machine intelligence version 0.9. 2, 2022- 06-27. Open Review62(1) (2022)

    2022

  13. [21]

    In: ICML

    Lee, K., Joshi, M., Turc, I.R., Hu, H., Liu, F., Eisenschlos, J.M., Khandelwal, U., Shaw, P., Chang, M.W., Toutanova, K.: Pix2struct: Screenshot parsing as pretrain- ing for visual language understanding. In: ICML. PMLR (2023)

    2023

  14. [22]

    In: ACL 2023 (2023)

    Liu, F., Eisenschlos, J., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., Altun, Y.: Deplot: One-shot visual language reasoning by plot-to-table translation. In: ACL 2023 (2023)

    2023

  15. [23]

    In: Proceedings of the 61st Annual Meeting of the ACL (Volume 1: Long Papers) (2023) 16 P

    Liu, F., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Altun, Y., Collier, N., Eisenschlos, J.: Matcha: Enhancing visual language pretraining with math rea- soning and chart derendering. In: Proceedings of the 61st Annual Meeting of the ACL (Volume 1: Long Papers) ...

    2023

  16. [24]

    arXiv preprint arXiv:1906.11906 (2019)

    Liu, X., Klabjan, D., NBless, P.: Data extraction from charts via single deep neural network. arXiv preprint arXiv:1906.11906 (2019)

    2019 arXiv

  17. [25]

    In: Proceedings of the IEEE/CVF WACV (2021)

    Luo, J., Li, Z., Wang, J., Lin, C.Y.: Chartocr: Data extraction from charts images via a deep hybrid framework. In: Proceedings of the IEEE/CVF WACV (2021)

    2021

  18. [26]

    In: ICDAR

    Ma, W., Zhang, H., Yan, S., Yao, G., Huang, Y., Li, H., Wu, Y., Jin, L.: Towards an efficient framework for data extraction from chart images. In: ICDAR. Springer (2021)

    2021

  19. [27]

    arXiv preprint arXiv:2305.14761 (2023)

    Masry, A., Kavehzadeh, P., Do, X.L., Hoque, E., Joty, S.: Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761 (2023)

    2023 arXiv

  20. [28]

    In: Proceedings of the 31stInternationalConferenceonComputationalLinguistics:IndustryTrack(2025)

    Masry, A., Thakkar, M., Bajaj, A., Kartha, A., Hoque, E., Joty, S.: Chartgemma: Visual instruction-tuning for chart reasoning in the wild. In: Proceedings of the 31stInternationalConferenceonComputationalLinguistics:IndustryTrack(2025)

    2025

  21. [29]

    Human Factors: The Journal of Human Factors and Ergonomics Society41(1999)

    Meyer,J.,Shamo,M.,Gopher,D.:Informationstructureandtherelativeefficacyof tables and graphs. Human Factors: The Journal of Human Factors and Ergonomics Society41(1999). https://doi.org/10.1518/001872099779656707, [Accessed 13-02- 2026]

    1999 doi

  22. [30]

    In: 2011 sixth international conference on digital information manage- ment

    Mishchenko, A., Vassilieva, N.: Chart image understanding and numerical data extraction. In: 2011 sixth international conference on digital information manage- ment. IEEE (2011)

    2011

  23. [31]

    https://plotdigitizer.com, [Accessed 19-01-2026]

    PlotDigitizer: PlotDigitizer: Extract Data from Graph Image Online — plotdigi- tizer.com. https://plotdigitizer.com, [Accessed 19-01-2026]

    2026

  24. [32]

    In: International Society for Music Information Retrieval Conference (2024)

    Riou, A., Lattner, S., Hadjeres, G., Anslow, M., Peeters, G.: Stem-jepa: A joint- embedding predictive architecture for musical stem compatibility estimation. In: International Society for Music Information Retrieval Conference (2024)

    2024

  25. [33]

    In: 2025 IEEE/CVF WACV

    Saito, A., Kudeshia, P., Poovvancheri, J.: Point-jepa: A joint embedding predic- tive architecture for self-supervised learning on point cloud. In: 2025 IEEE/CVF WACV. pp. 7348–7357. IEEE (2025)

    2025

  26. [34]

    In: Proceedings of the 24th annual ACM symposium on User interface software and technology (2011)

    Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: Revision: Automated classification, analysis and redesign of chart images. In: Proceedings of the 24th annual ACM symposium on User interface software and technology (2011)

    2011

  27. [35]

    IEEE Open Journal of the Computer Society4(2023)

    Shahira, K., Joshi, P., Lijiya, A.: Data extraction and question answering on chart images towards accessibility and data interpretation. IEEE Open Journal of the Computer Society4(2023)

    2023

  28. [36]

    arXiv preprint arXiv:2308.11788 (2023)

    Soto, C., Yoo, S.: An extensible point-based method for data chart value detection. arXiv preprint arXiv:2308.11788 (2023)

    2023 arXiv

  29. [37]

    arXiv preprint arXiv:2410.05016 (2024)

    Thimonier, H., Costa, J.L.D.M., Popineau, F., Rimmel, A., Doan, B.L.: T- jepa: Augmentation-free self-supervised learning for tabular data. arXiv preprint arXiv:2410.05016 (2024)

    2024 arXiv

  30. [38]

    arXiv preprint arXiv:2507.02915 (2025)

    Tuncay, L., Labbé, E., Benetos, E., Pellegrini, T.: Audio-jepa: Joint-embedding predictive architecture for audio representation learning. arXiv preprint arXiv:2507.02915 (2025)

    2025 arXiv

  31. [39]

    In: Interna- tionalSymposiumonInformationandCommunicationTechnology.Springer(2024)

    Vo, K.H., Nguyen, D.P., Nguyen, T.T., Quan, T.T.: Ti-jepa: An innovative energy- based joint embedding strategy for text-image multimodal systems. In: Interna- tionalSymposiumonInformationandCommunicationTechnology.Springer(2024)

    2024

  32. [40]

    In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A

    Xu, Y., Zhang, J., ZHANG, Q., Tao, D.: Vitpose: Simple vision transformer base- lines for human pose estimation. In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Sys- tems. vol. 35. Curran Associates, In...

    2022

  33. [41]

    arXiv preprint arXiv:2308.07743 (2023)

    Xue, W., Chen, D., Yu, B., Chen, Y., Zhou, S., Peng, W.: Chartdetr: A multi-shape detection network for visual chart recognition. arXiv preprint arXiv:2308.07743 (2023)

    2023 arXiv

  34. [42]

    Journal of Visualization24(2) (2021) A Parameters T able 5.Parameters used to generate our datasets

    Zhou,F.,Zhao,Y.,Chen,W.,Tan,Y.,Xu,Y.,Chen,Y.,Liu,C.,Zhao,Y.:Reverse- engineering bar charts using neural networks. Journal of Visualization24(2) (2021) A Parameters T able 5.Parameters used to generate our datasets. The colors are selected to be rich in contrast, to avoid whit...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.