Towards Polyglot Data Processing in Social Networks using the Hadoop-Spark ecosystem

Antony Seabra; Sergio Lifschitz

REVIEW 4 major objections 6 minor 16 references

Towards Polyglot Data Processing in Social Networks using the Hadoop-Spark ecosystem

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A polyglot Hadoop-Spark pipeline can rank influencers, count terms, and build follower graphs, with execution times that fall as tweet volume grows.

desk verdict A small, honest benchmark case study that overreaches on scalability; useful as a cautionary example of inferring capacity from wall-clock time. read the letter →

arxiv 2504.14314 v1 pith:FGQ2RATS submitted 2025-04-19 cs.DC

Antony Seabra , Sergio Lifschitz This is my paper · ORCID

classification cs.DC

keywords polyglotdataprocessingHadoopSparksocialnetworkanalysisHiveHBaseGraphXscalability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that social-network data processing should not rely on a single compute or storage engine. Instead, it proposes a polyglot approach within the Hadoop-Spark ecosystem, pairing each analytical task—ranking influential users, counting frequent terms, discovering user-group relationships—with the engine best suited to it: Hive or HBase for querying, MapReduce or Spark for term counting, GraphX for graph construction. To support this, the authors ran the three tasks on 500K and 5M tweet datasets and report average execution times of roughly 11 seconds and 16 seconds for the influence query, 28 seconds for term counting, and 42 seconds for graph generation on the smaller set, with lower times on the larger set. The paper's central claim is that the approach is feasible, efficient, and likely scalable, letting practitioners choose engines per task while keeping performance stable or improving as data grows.

What carries the argument

The load-bearing mechanism is Polyglot Data Processing: a combination of polyglot persistence (storing the same Twitter data simultaneously in HDFS, Hive external or internal tables, and HBase) and polyglot computing (choosing Spark, MapReduce, or GraphX per task). The paper treats HDFS as the central repository and ingestion landing zone, Hive as a SQL-like warehouse, HBase as a column-family NoSQL store for read-intensive queries, and Spark's GraphX as the in-memory graph layer. This machinery lets the authors attribute the observed performance to matching each task's data access pattern to the appropriate engine, rather than to any single technology.

What would settle it

Run the same three tasks on fresh clusters at 1M, 2M, and 10M tweets, recording per-run times and variance; if execution time rises at any step or varies widely across identical runs, the claimed scalability is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that a polyglot processing pipeline built from HDFS storage plus Hive, HBase, MapReduce, Spark, and GraphX can execute the three selected social-network tasks end to end, and that the measured average execution times for the 5M-tweet dataset were lower than those for the 500K-tweet dataset. The authors interpret this decrease as evidence that the solution is likely scalable. In their view, the experiment demonstrates that different engines have complementary strengths: Hive over HDFS handles the influence query in about 11 seconds versus about 16 seconds for HBase, MapReduce and Spark count frequent terms in about 28 seconds, and GraphX builds the user graph in about 42 seconds, with ingestion and storage under a minute for the smaller set and about seven minutes for the larger one.

Load-bearing premise

The scalability conclusion rests on comparing the 500K and 5M tweet runs as if they differ only in dataset volume; if the larger run benefited from a warmed-up cluster, cached data, different job plans, or random variation, the drop in execution time would not prove the system scales.

Editorial extensions

If this is right

Practitioners can build a social-media analytics pipeline where the same raw tweets in HDFS are queried through Hive for warehouse-style analysis and through HBase for column-oriented lookups without moving data between systems.
Term-frequency and influence-ranking tasks can be run with MapReduce or Spark over HDFS files, while graph tasks such as follower-community discovery use GraphX, and the choice per task is guided by measured execution time.
If the scalability observation holds, increasing the dataset volume should not require redesigning the pipeline; the same polyglot cycle of ingestion, storage, and evaluation can be repeated with larger collections.
The cyclical methodology gives a template for future benchmark studies: store data in multiple engines, run the same logical query on each, and compare average execution times to decide engine assignment.
The measured difference between Hive and HBase on the influence query suggests that for SQL-like aggregation over this workload, the warehouse engine should be preferred, and this kind of per-task comparison is the practical output of the approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported decrease in execution time from 500K to 5M tweets is likely driven by fixed overheads such as cluster startup, query planning, and JVM warm-up being amortized over more data, rather than by the engines becoming intrinsically faster; a fair scalability test would keep cluster state and cache conditions identical and report variance.
The paper's future-work list already points to Flink, Kafka, and Hudi; the untested implication is that the polyglot assignment rule could extend to streaming and incremental processing, routing real-time feeds through streaming engines while batch analytics continue on Hive and HBase.
The Hive-versus-HBase comparison is workload-specific: with different query shapes such as point lookups, scans, or updates, HBase's column-family storage could beat Hive, so the engine rankings should not be generalized beyond the three tasks.
A testable extension is to run the same three tasks at intermediate volumes of 1M, 2M, and 10M tweets and plot execution time against dataset size; the paper's scalability claim predicts flat or falling times, while an eventual rise would locate the crossover point.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You asked for a read on arXiv:2504.14314. My take: it's a workshop-level case study that runs three standard social-network analytics jobs across Hive, HBase, MapReduce, Spark, and GraphX on a three-node Dataproc cluster. That specific measurement combination is the only thing I'd call new. The polyglot framing comes from the literature they cite, and the tasks are textbook applications of each tool. The authors deserve credit for describing the setup and code clearly, and for not overselling what they did beyond one questionable inference. No self-citation, no equations to check, and the runs at least look real.

The soft spots are the ones the stress-test flags. The 5M-tweet execution times never appear in the text; we only get 'notably lower than initially expected' and a chart. No variance or per-trial values are given despite saying each job ran ten times. No controls for warm-up, caching, or JIT are mentioned. A monotone decrease in time as data grows 10x is the classic signature of a cold-start artifact, not scalable throughput. The paper itself says the result 'suggests' scalability and calls for further investigation, which is at least a hedged version, but the inference still doesn't follow from the data they show. Also, Task 1's SQL has a duplicated like_count in the influence sum—a minor but embarrassing bug. Task 2 compares MapReduce and Spark, but the results only mention MapReduce; the Spark number is missing. For Task 3 we get a graph build time but no comparison to anything else. The quantitative core is thin.

The reader's REJECT with moderate confidence is fair. The paper is not a serious research contribution on its own terms: the central empirical claim is unsupported, and the missing raw data makes it impossible to verify. That said, it is not incoherent and the authors are honestly reporting what they did. I would not send this to peer review at a journal or a competitive conference. It could be a decent workshop poster or a technical report, and the authors could strengthen it substantially by releasing measurements, adding error bars, controlling for cluster state, and fixing the SQL. As it stands, a serious editor should desk-reject rather than spend referee time on it.

For your own work: not something you'd cite, but worth a glance if you ever teach a class on benchmark pitfalls. I wouldn't bring it to reading group; the interesting conversation is about the methodology, not the results.

Referee Report

4 major / 6 minor

Summary. The paper proposes a 'polyglot data processing' approach within the Hadoop-Spark ecosystem, combining storage engines (HDFS, Hive, HBase) and processing frameworks (MapReduce, Spark, GraphX/GraphFrames) for three Twitter analytics tasks: identifying influential users, counting frequent terms, and building a user-follows graph. Experiments are conducted on a 3-node Google Dataproc cluster with datasets of approximately 500K and 5M tweets; execution times are averaged over ten runs. The authors conclude that the approach is efficient, feasible, and 'likely scalable' because execution times appeared to decrease from the smaller to the larger dataset.

Significance. If the empirical results were reliable, the paper would provide a useful practical benchmark for practitioners selecting Hadoop/Spark components for social-network analytics, and the task-to-engine mapping is a reasonable engineering contribution. The paper has strengths: it uses real Twitter data, describes a clear evaluation cycle, reports a concrete cluster configuration, and makes its workloads explicit. However, the quantitative core is currently too weak to support the stated efficiency and scalability claims: the 5M-dataset timings are missing, no variability is reported, experimental confounds (caching, warm-up, dataset composition) are uncontrolled, and the Task 1 influence query contains a formula error. The soundest part is the qualitative mapping of tasks to ecosystem tools, which may retain value if the quantitative claims are substantially revised or removed.

major comments (4)

[5.4, Figure 6] The scalability inference is presented in Section 5.4: 'The observation that execution times decrease as the dataset volume increases suggests that the solution is likely scalable.' Yet the execution times for the 5M-tweet dataset are never reported numerically; the text only says they were 'notably lower than initially expected' and points to Figure 6, which is not legible in the manuscript as provided. Without the actual values, the reader cannot verify the direction or magnitude of the change, so the central quantitative claim is unsubstantiated.
[5.4] Section 5.4 states that each job was executed ten times and that the reported times are averages, but no variance, per-trial measurements, confidence intervals, or statistical tests are provided. With n=10 and no dispersion information, the comparison between the 500K and 5M timings could easily be within run-to-run noise, so the observed difference does not constitute evidence of scalability.
[5.4] The experimental design does not control for cluster warm-up, OS page cache effects, JVM JIT compilation, HDFS block caching, or differences in tweet composition between the two datasets. A monotone decrease in execution time with a 10x larger dataset is the classic signature of a caching or warm-up artifact, and the paper's own expression of surprise ('notably lower than initially expected') indicates that the authors did not expect this outcome. The conclusion that the solution is likely scalable therefore exceeds what the measurements can support.
[5.1, Task 1 query] The SQL query in Section 5.1 defines influence as `SUM(impression_count) + SUM(like_count) + SUM(like_count) + SUM(reply_count) + SUM(retweet_count)`, which double-counts `like_count` and omits `quote_count`, contradicting the task definition in Section 4.1 that influence is based on impressions, likes, quotes, replies, and retweets. This error means the reported Task 1 results do not actually measure the stated influence metric.

minor comments (6)

[Figure 1 caption] The caption of Figure 1 contains a typo: 'Understanting' should be 'Understanding.'
[5.1, SQL query] The query begins with 'SELECT author_id.' where a comma after the column list is intended; as printed, the query is not syntactically valid.
[5.3] The text says that all graphs are generated using 'the Spark GraphX library and GraphFrames extension,' but the code fragment only uses GraphFrames; GraphX and GraphFrames are distinct APIs, and the manuscript should clarify which one was used.
[5.2] The Python code snippet contains non-ASCII characters inside the `replace` calls and omits the surrounding Spark session and stopword definitions, making the snippet hard to reproduce.
[5.4] The phrase 'the above-mentioned findings' is vague; the results should be anchored to specific figure numbers and, better, to a table with numeric execution times.
[References] The reference to Dean and Ghemawat (2004) lacks a publication venue, and the Apache Spark RDD reference points to a tutorialspoint URL rather than an authoritative specification; both should be updated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports direct measured execution times and draws an empirical scalability inference, with no fitted parameters, no definitional identities, and no load-bearing self-citation.

full rationale

This paper contains no derivation chain that could reduce to its own inputs. The central evidence is direct measurement of execution times on a live Hadoop-Spark cluster (Sections 5.1-5.4), with tasks defined independently of the results: influence ranking by aggregation over tweet metrics, term frequency counting via MapReduce/Spark transformations, and graph construction using GraphFrames. No parameter is fitted to the data and then renamed as a prediction; no quantity is defined in terms of the outcome it is said to explain. The scalability remark in Section 5.4, namely that 'execution times decrease as the dataset volume increases suggests that the solution is likely scalable,' is an empirical interpretation of measured run times, not an equation, a definitional identity, or a fitted result. Even though the 5M-dataset execution times are not reported numerically and the comparison may be affected by warm-up, caching, or other experimental confounds, that is a correctness and experimental-design concern, not circularity. Citations in the paper are to external Hadoop/Spark literature and prior polyglot-persistence work; none is a self-citation carrying the weight of the empirical claim. The conclusion that the polyglot approach enables choosing appropriate engines is a qualitative summary of the executed benchmarks rather than a circular derivation. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters, fitted constants, or invented entities appear. The paper's load-bearing assumptions are about the validity of the benchmark setup and the stability of the measurements.

assumptions (4)

domain assumption The Twitter datasets collected for 'The Last of Us' are representative of the social-network processing tasks being benchmarked.
Used in Section 4 to define the workload; no evidence is given that results generalize to other topics or network structures.
domain assumption Average execution time over ten runs is a stable performance measure without reporting variance or controlling warm-up and caching.
Section 5.4 reports only averages; all comparative claims depend on this stability.
domain assumption The Google Dataproc cluster performance is constant across the 500K and 5M experiments.
Sections 4 and 5 describe a single three-node cluster but give no evidence that JVM state, HDFS cache, or other system effects are identical between the two dataset runs.
domain assumption GraphFrame construction time in Task 3 measures the social-relationship analysis task.
Section 5.3 only builds a GraphFrame; no algorithm such as connected components or PageRank is executed, so the reported 42 seconds may not correspond to the stated analytical task.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Polyglot Data Processing in Social Networks using the Hadoop-Spark ecosystem." pith.science (2026). https://pith.science/paper/FGQ2RATS

@misc{pith2026250414314,
  author       = {Pith},
  title        = {Pith review of: Towards Polyglot Data Processing in Social Networks using the Hadoop-Spark ecosystem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGQ2RATS}},
  note         = {Machine review of arXiv:2504.14314}
}

read the original abstract

This article explores the use of the Hadoop-Spark ecosystem for social media data processing, adopting a polyglot approach with the integration of various computation and storage technologies, such as Hive, HBase and GraphX. We discuss specific tasks involved in processing social network data, such as calculating user influence, counting the most frequent terms in messages and identifying social relationships among users and groups. We conducted a series of empirical performance assessments, focusing on executing selected tasks and measuring their execution time within the Hadoop-Spark cluster. These insights offer a detailed quantitative analysis of the performance efficiency of the ecosystem tools. We conclude by highlighting the potential of the Hadoop-Spark ecosystem tools for advancing research in social networks and related fields.

Figures

Figures reproduced from arXiv: 2504.14314 by the authors.

**Figure 1.** Understanting MapReduce with Hadoop [Apache 2004] each key, which results in the number of occurrences of each word in the original input. Finally, the outputs of each Reducer are combined to generate the ultimate result. 2.3. Spark MapReduce jobs can be really complex and require many linked Map and Reduce tasks, so many files will have to be generated at runtime. As these files are generated on disk, processing be… view at source ↗

**Figure 2.** Apache Spark Resilient Distributed Datasets [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Evaluation cycle 4.1. Selected Tasks In this research, we use data sourced from the Twitter social network. The first task involves determining the most influential users within a specified scope. For each user, we calculate their influence through the number of impressions, likes, quotes, replies and retweets recorded for each posted tweet. The second task in our evaluation focuses on determining the frequency of t… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Polyglot Architecture In order to leverage the capabilities of Hive and HBase storage engines, the preprocessed data must be imported into their respective structures, enabling seamless integration and use within these frameworks. The decision to store the tweets and…

**Figure 5.** Figure 5: Execution times for the 500K dataset As outlined within the methodological framework, our approach involves a systematic iteration through the stages of ingestion, storage, and processing. We begin this iterative process with the 500,000 tweets dataset. Each job is e…

**Figure 6.** Figure 6: Execution times for the 5M dataset The second dataset utilized in this study consisted of a sizeable collection of 5,000,000 (five million) tweets. The process of ingesting and storing this dataset was executed in an approximate duration of 7 minutes. The obtained re…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

Understanding mapreduce with hadoop

Apache (2004). Understanding mapreduce with hadoop. https://cwiki.apache.org/confluence/download/attachments/120729986/MapReduce-SPA2008.pdf?version=1&modificationDate=1562667747000&api=v2. Accessed: 2024-01-20

arXiv 2004
[3]

Aziz, K., Zaidouni, D., and Bellafkih, M. (2018). Real-time data analysis using spark and hadoop. In 2018 4th international conference on optimization and applications (ICOA) , pages 1--6. IEEE

work page 2018
[4]

and Ghemawat, S

Dean, J. and Ghemawat, S. (2004). Mapreduce: Simplified data processing on large clusters

work page 2004
[5]

and Kaur, D

Garg, K. and Kaur, D. (2019). Sentiment analysis on twitter data using apache hadoop and performance evaluation on hadoop mapreduce and apache spark. In Proceedings on the International Conference on Artificial Intelligence (ICAI) , pages 233--238. The Steering Committee of The World Congress in Computer Science

work page 2019
[6]

Ghemawat, S., Gobioff, H., and Leung, S.-T. (2003). The google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles , pages 29--43

work page 2003
[7]

Glake, D., Kiehn, F., Schmidt, M., Panse, F., and Ritter, N. (2022). Towards polyglot data stores--overview and open research questions. arXiv preprint arXiv:2204.05779

arXiv 2022
[8]

Karun, A. K. and Chitharanjan, K. (2013). A review on hadoop hdfs infrastructure extensions. In 2013 IEEE conference on information & communication technologies , pages 132--137. IEEE

work page 2013

Show all 16 references

[9]

Nandimath, J., Banerjee, E., Patil, A., Kakade, P., Vaidya, S., and Chaturvedi, D. (2013). Big data analysis using apache hadoop. In 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI) , pages 700--703. IEEE

2013
[10]

S., Sinha, V., and Singh, N

Sachdeva, K., Lamba, J. S., Sinha, V., and Singh, N. (2016). Comparison of data processing tools in hadoop. In 2016 International Conference on Electrical, Electronics, Communication, Computer and Optimization Techniques (ICEECCOT) , pages 238--242. IEEE

2016
[11]

Sadalage, P. J. and Fowler, M. (2013). NoSQL distilled: a brief guide to the emerging world of polyglot persistence . Pearson Education

2013
[12]

and Agarwal, A

Sehgal, D. and Agarwal, A. K. (2016). Sentiment analysis of big data applications using twitter data with the help of hadoop framework. In 2016 international conference system modeling & advancement in research trends (SMART) , pages 251--255. IEEE

2016
[13]

Sheela, L. J. (2016). A review of sentiment analysis in twitter data using hadoop. International Journal of Database Theory and Application , 9(1):77--86

2016
[14]

Apache spark resilient distributed datasets

Spark (2013). Apache spark resilient distributed datasets. https://www.tutorialspoint.com/apache_spark/apache_spark_rdd.htm. Accessed: 2024-01-20

2013
[15]

Yang, D., Yang, H., Wang, P., and Li, S. (2017). Design and implement of large-scale social network analysis platform based on hadoop. J. Inf. Hiding Multim. Signal Process. , 8(2):300--309

2017
[16]

J., Shenker, S., and Stoica, I

Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauly, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Presented as part of the 9th \ USENIX \ Symposium on Networ...

2012

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

Understanding mapreduce with hadoop

Apache (2004). Understanding mapreduce with hadoop. https://cwiki.apache.org/confluence/download/attachments/120729986/MapReduce-SPA2008.pdf?version=1&modificationDate=1562667747000&api=v2. Accessed: 2024-01-20

arXiv 2004

[3] [3]

Aziz, K., Zaidouni, D., and Bellafkih, M. (2018). Real-time data analysis using spark and hadoop. In 2018 4th international conference on optimization and applications (ICOA) , pages 1--6. IEEE

work page 2018

[4] [4]

and Ghemawat, S

Dean, J. and Ghemawat, S. (2004). Mapreduce: Simplified data processing on large clusters

work page 2004

[5] [5]

and Kaur, D

Garg, K. and Kaur, D. (2019). Sentiment analysis on twitter data using apache hadoop and performance evaluation on hadoop mapreduce and apache spark. In Proceedings on the International Conference on Artificial Intelligence (ICAI) , pages 233--238. The Steering Committee of The World Congress in Computer Science

work page 2019

[6] [6]

Ghemawat, S., Gobioff, H., and Leung, S.-T. (2003). The google file system. In Proceedings of the nineteenth ACM symposium on Operating systems principles , pages 29--43

work page 2003

[7] [7]

Glake, D., Kiehn, F., Schmidt, M., Panse, F., and Ritter, N. (2022). Towards polyglot data stores--overview and open research questions. arXiv preprint arXiv:2204.05779

arXiv 2022

[8] [8]

Karun, A. K. and Chitharanjan, K. (2013). A review on hadoop hdfs infrastructure extensions. In 2013 IEEE conference on information & communication technologies , pages 132--137. IEEE

work page 2013