Large Scale Organization and Inference of an Imagery Dataset for Public Safety

Andrew Weinert; David Strohschein; Jeffrey Liu; Siddharth Samsi

REVIEW 3 major objections 5 minor 16 references

Large Scale Organization and Inference of an Imagery Dataset for Public Safety

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Packing public-safety imagery into monthly HDF5 chunks and indexing it by content hash makes terabyte-scale cluster inference practical, as shown by two classifier runs covering 458,000 Civil Air Patrol images in 153 node-hours.

desk verdict Workshop-quality engineering report with a genuinely useful new dataset; the efficiency claim is plausible but lacks a baseline, so treat it as conditional. read the letter →

arxiv 1908.09006 v1 pith:HXDTIWXM submitted 2019-08-16 cs.CV cs.LGeess.IVstat.ML

Jeffrey Liu , David Strohschein , Siddharth Samsi , Andrew Weinert This is my paper · ORCID

classification cs.CVcs.LGeess.IVstat.ML

keywords publicsafetyimagerydatasetHDF5LustrefilesystemAccumuloD4MassociativearraydistributedinferenceCivilAirPatrolimageclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors assembled a multi-terabyte imagery and video collection for public-safety computer vision from operational sources and open video, then claim that a hierarchical organization makes this data usable on a shared high-performance cluster. Raw files are archived into large split archives, serialized into monthly HDF5 chunks that preserve the original directory structure, and indexed in a content-hashed NoSQL associative array, so a gigabyte-scale index stands in for terabytes of raw data. They validate the arrangement by running two pretrained classifiers over the 458,000-image Civil Air Patrol subset, reporting 60.4 node-hours for ImageNet Inception-ResNetV2 and 92.7 node-hours for Places365 ResNet50, including file loading and preprocessing. The reader should care because public-safety analytics are expected to stress the nationwide broadband network, yet few operational datasets exist to develop those capabilities.

What carries the argument

The load-bearing mechanism is a two-part storage-and-index design. First, data is packed into large HDF5 files, each an independent monthly chunk organized like a Linux directory tree with Metadata and Annotations stored as JSON. Large files are chosen deliberately to exploit the cluster file system's striping and 1 MB block size; read-only opening and single-process metadata listing are meant to keep metadata-server traffic low. Second, a D4M associative array over an Accumulo NoSQL backend indexes each file by its SHA1 hash, using hierarchical columns of the form Type | Source | Field, so terabytes of raw content are represented by gigabytes of index entries. The inference harness that consumes this layout is the third component: 32 nodes running pretrained ImageNet and Places365 classifiers over the chunked Civil Air Patrol data, with runtimes accumulated into node-hours.

What would settle it

Run the same 458,000-image inference job on the same cluster with the dataset stored as ordinary small image files instead of HDF5 chunks, keeping the models and node count fixed. If the small-file layout matches or beats the reported 60.4 and 92.7 node-hour totals without metadata-server degradation, the paper's central architectural claim fails; if metadata traffic saturates and runtime grows sharply, the claim is confirmed.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that a terabyte-scale public-safety imagery collection does not have to be a chaotic pile of small files to be ready for machine learning. The proposed organization has three layers: raw data archived in 4.5 GB split tar files; a serialized layer of HDF5 files, each covering one month and mirroring the original file layout, with metadata and annotations carried as JSON; and an index built as an associative array in an Accumulo NoSQL database, with rows keyed by SHA1 content hash and hierarchical columns specifying type, source, and field. This lets queries resolve to specific HDF5 chunks without scanning raw data, and new columns or labels can be appended without a schema change. As evidence, the authors ran distributed inference on 32 GPU nodes over the Civil Air Patrol subset, where each node handled roughly 14,351 images; ImageNet Inception-ResNetV2 took an average of 113.25 minutes per node (60.4 node-hours total) and Places365 ResNet50 took 173.74 minutes per node (92.7 node-hours total). These runs are offered as proof that the organized dataset supports large-scale inference across terabytes.

Load-bearing premise

The efficiency case rests on the unmeasured premise that millions of small image files would overwhelm the cluster's metadata servers, so packing data into large HDF5 chunks is necessary; no baseline comparison with small-file storage is included.

Editorial extensions

If this is right

A researcher targeting a known event can download only the monthly HDF5 chunks for that time window rather than the entire multi-terabyte dataset.
New data and labels can be added incrementally: each month becomes a new HDF5 chunk, and new columns in the index require no schema migration.
The reported node-hour totals give public-safety developers a concrete planning figure for large-scale image classification on a modest GPU allocation.
Machine-generated annotations already in the index support content-based search and retrieval even before human labels are used.
The planned transfer of the dataset to a federal agency makes the organized collection available to the broader public-safety research community, addressing the gap identified in the government roadmap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Amortizing the reported node-hours over 458,000 images gives roughly 0.5 seconds per image for the ImageNet run and 0.7 seconds per image for the Places run; the paper does not state these derived rates, but they could serve as planning heuristics for similar jobs.
The same chunked-HDF5 plus content-hashed index recipe should generalize to other continually growing image collections such as body-camera footage or drone surveillance, although the paper tests only one cluster and one workload.
Because rows are keyed by SHA1 content hash, the index could support near-duplicate detection and change monitoring across the growing dataset, a use the paper does not explore.
A direct A/B comparison against ordinary small-file storage, which the authors did not run, would determine whether the Lustre metadata bottleneck is real or whether a simpler organization would suffice; until that comparison exists, the central efficiency claim depends on an architectural assumption.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plain take: this is a workshop-quality engineering report, not a research advance, but the dataset resource is real and the design is sensible. What's actually new: the aggregation of operational and representative public-safety imagery from CAP, DVIDS, MA-TF1, traffic cams, URSA, USGS, and YouTube into one multi-terabyte corpus, with monthly HDF5 chunks and a D4M/Accumulo index. That corpus is a genuinely useful contribution for the public-safety vision community, and the paper gives enough architectural detail to reproduce the layout. The node-hour arithmetic checks out (32 nodes × 113.25 min ≈ 60.4 node-hours, and similarly for Places), and the writing is honest about deferring annotation details to another paper and about using prior LLSC components. Credit is due for not overclaiming the science.

Soft spots, in order of size. The main claim is that the data was 'intelligently organized' for efficient compute and storage, but the paper never tests that claim. Section II-B asserts that many small files would overwhelm Lustre's MDS and that large HDF5 chunks fix it, but no baseline compares the HDF5 layout against a small-file directory or TFRecords on the same cluster. The end-to-end inference runtimes in Section IV include file loading, preprocessing, and CNN compute, so they don't isolate I/O; if the same run on ordinary JPEGs gave comparable wall-clock times, the architectural premise would be unsupported. That is a completeness gap, not an inconsistency. Also, the CPU inference run is mentioned but never reported, and the dataset and code are not yet public, which limits verification. Minor: some references are self-citations to earlier LLSC work, but they are the right technical sources for HDF5/D4M, so that's not a real flaw.

Who it's for: engineers standing up large image corpora on HPC clusters, and public-safety CV researchers who want the dataset. It deserves a serious referee, but a referee should push for an I/O baseline, the CPU numbers, and a release plan. My verdict is conditional acceptance rather than rejection; the core description is credible.

Referee Report

3 major / 5 minor

Summary. The paper describes a hierarchical organization scheme for a multi-terabyte public safety imagery and video dataset stored on the Lincoln Laboratory Supercomputing Cluster (LLSC). Raw data are archived as 4.5 GB tar splits; imagery and video are serialized into month-level HDF5 files, each containing a data group that mirrors the original directory structure plus JSON metadata and annotation groups; and a D4M/Accumulo associative-array index supports content-based queries. The authors then report end-to-end GPU inference runs over the 458,000-image Civil Air Patrol subset using two pretrained classifiers: Inception-ResNetV2 on ImageNet (60.4 node-hours) and ResNet50 on Places365 (92.7 node-hours), stating that CPU inference on KNL nodes was also run. The paper's central claim is that this organization enables 'efficient compute and storage' for large-scale imagery inference.

Significance. If the organizational design were validated by measurements, this paper would be a useful systems/experience report for researchers who need to run large-scale vision inference on HPC clusters with Lustre storage. The reported node-hour arithmetic is internally consistent and the runtime measurements are transparent about including loading and preprocessing. The paper also names concrete open-source components (HDF5, D4M, Accumulo) and states that software is released on GitHub. However, the central efficiency claim is currently an assertion: no baseline comparison is provided, no I/O time is isolated, and the paper's own text defers annotation details to an uncited companion paper. The significance is therefore conditional on supplying the missing measurements or substantially tempering the claim.

major comments (3)

[Section II-B] The efficiency premise is load-bearing but unmeasured. The paper asserts that storing AI data as many small files is 'unsuitable' on Lustre because of MDS serialization and network traffic, and therefore uses large HDF5 files. No experiment in this paper compares the chosen HDF5 layout against the original small-file layout, against TFRecords, or against direct reads of JPEG files on the same cluster. Furthermore, the Section IV runtimes are end-to-end CNN timings that include loading and preprocessing but do not isolate I/O time, so they cannot distinguish the benefit of the file organization from the cost of GPU compute. I recommend adding a small-scale I/O benchmark (e.g., reading 10k images from each layout) or an explicit comparison on a subset, or revising the 'intelligently organized' claim to a descriptive one.
[Section IV] The sentence 'We also ran the inference task on 32 KNL CPU nodes' reports an experiment that is never described or quantified anywhere in the paper. Additionally, only the average runtime per node is given for the GPU runs; there are no node-level statistics, standard deviations, or min/max values, so load imbalance and variance across the 32 nodes cannot be assessed. Please either report the CPU results and per-node distribution or remove the sentence and add variance information for the GPU runs.
[Section III-B] The annotation subsection ends with 'Details on the annotations can be found in another paper' but gives no citation. Since human and machine annotations are a central component of the dataset and of the Accumulo index described in Section III-E, this omission makes the annotation pipeline unverifiable in the present manuscript. Please cite the companion paper or provide a self-contained description of the annotation process and label schema.

minor comments (5)

[Abstract/Introduction] There are several typographical errors, including 'represenative' in the Introduction and 'high performace computing' in Section II-B; a proofreading pass is needed.
[Section II-B] The sentence 'Since the block size of Lustre of is 1MB' contains a garbled phrase ('of is'); it should read 'Since the block size of Lustre is 1MB'.
[Index Terms] The index terms 'public safety,video' are missing a space after the comma; they should read 'public safety, video'.
[Section III-A] Table I lists 'Massachusetts traffic cameras Images 150,000' without indicating the source organization or collection method; a brief footnote would improve reproducibility.
[Section IV] The runtimes are reported in minutes per node with two decimal places, but the number of images per node is given only as an average (14,351). Reporting the total image count and the per-node load distribution would make the node-hour arithmetic easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports system design and external model inference; node-hour values are measured arithmetic, not derived from fitted parameters.

full rationale

The paper's central claims are (i) an organizational scheme for a multi-terabyte public safety imagery dataset and (ii) large-scale inference results over the CAP subset. The reported node-hour figures are direct products of measured per-node runtimes multiplied by node counts; no equation derives a result from an input in a way that presupposes the result, and no fitted parameter is renamed as a prediction. The HDF5-chunking recommendation is asserted from LLSC experience rather than benchmarked, and the paper contains self-citations to MIT-LL authored D4M/Accumulo and Lustre references, but these are not load-bearing in a circular sense: the Lustre MDS discussion is standard external architectural knowledge, and D4M is an implementation choice rather than a claimed derivation. Inference is evaluated with external pretrained models (ImageNet Inception-ResNetV2 and Places365 ResNet50), so the measured runtimes are against independent benchmarks. The absence of a baseline comparing HDF5 chunks against small-file storage is an experimental completeness/correctness risk, not circular reasoning. No uniqueness theorem is invoked, and no target result is assumed by construction. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new postulates or fitted numbers. Its central claim depends on established but unmeasured system-design assumptions about Lustre, HDF5, and Accumulo, plus the representativeness of pretrained classifiers for public safety imagery.

assumptions (5)

domain assumption Many small files cause significant metadata bottlenecks on Lustre
Invoked in Section II-B to justify large-file HDF5 organization; not measured in this paper.
domain assumption Large files (>100s of MB) improve parallel I/O performance on Lustre
Section II-B recommends HDF5/TFRecords to maximize file I/O performance; no benchmark is supplied.
domain assumption Pretrained ImageNet and Places365 classifiers are suitable for public safety imagery analysis
Section IV applies these models without task-specific accuracy evaluation.
domain assumption Monthly temporal chunking supports intuitive event-based data access
Section III-D assumes temporal organization aids researchers looking for specific events like hurricanes.
domain assumption Accumulo/D4M can scalably index the dataset with no a priori query optimization
Section III-E relies on cited D4M/Accumulo properties; no index performance measurements are given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Scale Organization and Inference of an Imagery Dataset for Public Safety." pith.science (2026). https://pith.science/paper/HXDTIWXM

@misc{pith2026190809006,
  author       = {Pith},
  title        = {Pith review of: Large Scale Organization and Inference of an Imagery Dataset for Public Safety},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXDTIWXM}},
  note         = {Machine review of arXiv:1908.09006}
}

read the original abstract

Video applications and analytics are routinely projected as a stressing and significant service of the Nationwide Public Safety Broadband Network. As part of a NIST PSCR funded effort, the New Jersey Office of Homeland Security and Preparedness and MIT Lincoln Laboratory have been developing a computer vision dataset of operational and representative public safety scenarios. The scale and scope of this dataset necessitates a hierarchical organization approach for efficient compute and storage. We overview architectural considerations using the Lincoln Laboratory Supercomputing Cluster as a test architecture. We then describe how we intelligently organized the dataset across LLSC and evaluated it with large scale imagery inference across terabytes of data.

Figures

Figures reproduced from arXiv: 1908.09006 by the authors.

**Figure 1.** Lustre architecture. The MDS manages all name space operations for a Lustre file system. A file system’s directory hierarchy and file information are contained on storage devices referred to as Metadata Targets (MDT), and the MDS provides the logical interface to this storage. OSSs provide bulk storage for the contents of files in a Lustre file system. One or more object storage servers (OSS) store file data on one… view at source ↗

**Figure 2.** Example images included in the dataset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Example HDF5 individual file organization. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Example HDF5 chunking. E. Data Indexing with Accumulo The dataset is indexed using an associative array including metadata and annotations to facilitate searching through the data. Queries can be performed against the index to locate data of interest, e.g. to identify …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[1]

Public Safety Analytics R&D Roadmap,

T. McElvaney, R. Felts, and M. Leh, “Public Safety Analytics R&D Roadmap,” NIST, NIST Technical Note 1917, Apr. 2016. [Online]. Available: https://doi.org/10.6028/NIST.TN.1917

work page doi:10.6028/nist.tn.1917 1917
[2]

Outreach to Deﬁne a Public Safety Com- munications Model For Broadband Cellular Video,

A. Weinert and C. Budny, “Outreach to Deﬁne a Public Safety Com- munications Model For Broadband Cellular Video,” in 2018 IEEE International Symposium on Technologies for Homeland Security (HST) . Woburn, MA: IEEE, Oct. 2018, pp. 1–4

work page 2018
[3]

A Vision for Technology-mediated Support for Public Participation & Assistance in Mass Emergencies & Disasters,

L. Palen, K. M. Anderson, G. Mark, J. Martin, D. Sicker, M. Palmer, and D. Grunwald, “A Vision for Technology-mediated Support for Public Participation & Assistance in Mass Emergencies & Disasters,” in Proceedings of the 2010 ACM-BCS Visions of Computer Science Conference , ser. ACM-BCS ’10. Swinton, UK, UK: British Computer Society, 2010, pp. 8:1–8:12. [...

work page arXiv 2010
[4]

First workshop on video analytics in public safety,

J. S. Garofolo, S. L. Garﬁnkel, and R. B. Schwartz, “First workshop on video analytics in public safety,” NIST, NIST Pubs NISTR- 8164, Jan. 2017. [Online]. Available: https://www.nist.gov/publications/ ﬁrst-workshop-video-analytics-public-safety

work page 2017
[5]

Y ouTube-8m: A Large-Scale Video Classiﬁcation Benchmark,

S. Abu-El-Haija, N. Kothari, J. Lee, P . Natsev, G. Toderici, B. V aradarajan, and S. Vijayanarasimhan, “Y ouTube-8m: A Large-Scale Video Classiﬁcation Benchmark,” arXiv:1609.08675 [cs] , Sep. 2016, arXiv: 1609.08675. [Online]. Available: http://arxiv.org/abs/1609.08675

arXiv 2016
[6]

Lustre, hadoop, accumulo,

J. Kepner, W. Arcand, D. Bestor, B. Bergeron, C. Byun, L. Edwards, V . Gadepally, M. Hubbell, P . Michaleas, J. Mullen, A. Prout, A. Rosa, C. Y ee, and A. Reuther, “Lustre, hadoop, accumulo,” in 2015 IEEE High Performance Extreme Computing Conference (HPEC) , Sep. 2015, pp. 1–5

work page 2015
[7]

Efﬁciently Scaling Up Video Annotation with Crowdsourced Marketplaces,

C. V ondrick, D. Ramanan, and D. Patterson, “Efﬁciently Scaling Up Video Annotation with Crowdsourced Marketplaces,” in Computer Vision â ˘A¸ S ECCV 2010 , ser. Lecture Notes in Computer Science, K. Daniilidis, P . Maragos, and N. Paragios, Eds. Springer Berlin Heidelberg, 2010, pp. 610–623

work page 2010
[8]

Incentivizing High Quality Crowdwork,

C.-J. Ho, A. Slivkins, S. Suri, and J. W. V aughan, “Incentivizing High Quality Crowdwork,” in Proceedings of the 24th International Conference on World Wide Web , ser. WWW ’15. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2015, pp. 419–429, event-place: Florence, Italy. [Online]. Available: http...

arXiv 2015

Show all 16 references

[9]

ImageNet Large Scale Visual Recognition Challenge,

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015

2015
[10]

Places: A 10 Million Image Database for Scene Recognition,

B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 Million Image Database for Scene Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 6, pp. 1452–1464, Jun. 2018. [Online]. Available: https://ieeexplore.ieee.org...

2018
[11]

An overview of the HDF5 technology suite and its applications,

M. Folk, G. Heber, Q. Koziol, E. Pourmal, and D. Robinson, “An overview of the HDF5 technology suite and its applications,” in Proceed- ings of the EDBT/ICDT 2011 Workshop on Array Databases , Uppsala, Sweden, 2011, pp. 36–47

2011
[12]

Building Damage Assessment Using Deep Learning and Ground-Level Image Data,

K. R. Nia and G. Mori, “Building Damage Assessment Using Deep Learning and Ground-Level Image Data,” in 2017 14th Conference on Computer and Robot Vision (CRV) , May 2017, pp. 95–102

2017
[13]

Dynamic distributed dimensional data model (D4m) database and computation system,

J. Kepner, W. Arcand, W. Bergeron, N. Bliss, R. Bond, C. Byun, G. Condon, K. Gregson, M. Hubbell, J. Kurz, A. McCabe, P . Michaleas, A. Prout, A. Reuther, A. Rosa, and C. Y ee, “Dynamic distributed dimensional data model (D4m) database and computation system,” in 2012 IEEE Int...

2012
[14]

D4m 2.0 schema: A general purpose high performance schema for the Accumulo database,

J. Kepner, C. Anderson, W. Arcand, D. Bestor, B. Bergeron, C. Byun, M. Hubbell, P . Michaleas, J. Mullen, D. O’Gwynn, A. Prout, A. Reuther, A. Rosa, and C. Y ee, “D4m 2.0 schema: A general purpose high performance schema for the Accumulo database,” in 2013 IEEE High Performanc...

2013
[15]

High-throughput ingest of data provenance records into Accumulo,

T. Moyer and V . Gadepally, “High-throughput ingest of data provenance records into Accumulo,” in 2016 IEEE High Performance Extreme Computing Conference (HPEC) , Sep. 2016, pp. 1–6

2016
[16]

Driving big data with big compute,

C. Byun, W. Arcand, D. Bestor, B. Bergeron, M. Hubbell, J. Kepner, A. McCabe, P . Michaleas, J. Mullen, D. O’Gwynn, A. Prout, A. Reuther, A. Rosa, and C. Y ee, “Driving big data with big compute,” in 2012 IEEE Conference on High Performance Extreme Computing , Sep. 2012, pp. 1–6

2012

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Public Safety Analytics R&D Roadmap,

T. McElvaney, R. Felts, and M. Leh, “Public Safety Analytics R&D Roadmap,” NIST, NIST Technical Note 1917, Apr. 2016. [Online]. Available: https://doi.org/10.6028/NIST.TN.1917

work page doi:10.6028/nist.tn.1917 1917

[2] [2]

Outreach to Deﬁne a Public Safety Com- munications Model For Broadband Cellular Video,

A. Weinert and C. Budny, “Outreach to Deﬁne a Public Safety Com- munications Model For Broadband Cellular Video,” in 2018 IEEE International Symposium on Technologies for Homeland Security (HST) . Woburn, MA: IEEE, Oct. 2018, pp. 1–4

work page 2018

[3] [3]

A Vision for Technology-mediated Support for Public Participation & Assistance in Mass Emergencies & Disasters,

L. Palen, K. M. Anderson, G. Mark, J. Martin, D. Sicker, M. Palmer, and D. Grunwald, “A Vision for Technology-mediated Support for Public Participation & Assistance in Mass Emergencies & Disasters,” in Proceedings of the 2010 ACM-BCS Visions of Computer Science Conference , ser. ACM-BCS ’10. Swinton, UK, UK: British Computer Society, 2010, pp. 8:1–8:12. [...

work page arXiv 2010

[4] [4]

First workshop on video analytics in public safety,

J. S. Garofolo, S. L. Garﬁnkel, and R. B. Schwartz, “First workshop on video analytics in public safety,” NIST, NIST Pubs NISTR- 8164, Jan. 2017. [Online]. Available: https://www.nist.gov/publications/ ﬁrst-workshop-video-analytics-public-safety

work page 2017

[5] [5]

Y ouTube-8m: A Large-Scale Video Classiﬁcation Benchmark,

S. Abu-El-Haija, N. Kothari, J. Lee, P . Natsev, G. Toderici, B. V aradarajan, and S. Vijayanarasimhan, “Y ouTube-8m: A Large-Scale Video Classiﬁcation Benchmark,” arXiv:1609.08675 [cs] , Sep. 2016, arXiv: 1609.08675. [Online]. Available: http://arxiv.org/abs/1609.08675

arXiv 2016

[6] [6]

Lustre, hadoop, accumulo,

J. Kepner, W. Arcand, D. Bestor, B. Bergeron, C. Byun, L. Edwards, V . Gadepally, M. Hubbell, P . Michaleas, J. Mullen, A. Prout, A. Rosa, C. Y ee, and A. Reuther, “Lustre, hadoop, accumulo,” in 2015 IEEE High Performance Extreme Computing Conference (HPEC) , Sep. 2015, pp. 1–5

work page 2015

[7] [7]

Efﬁciently Scaling Up Video Annotation with Crowdsourced Marketplaces,

C. V ondrick, D. Ramanan, and D. Patterson, “Efﬁciently Scaling Up Video Annotation with Crowdsourced Marketplaces,” in Computer Vision â ˘A¸ S ECCV 2010 , ser. Lecture Notes in Computer Science, K. Daniilidis, P . Maragos, and N. Paragios, Eds. Springer Berlin Heidelberg, 2010, pp. 610–623

work page 2010

[8] [8]

Incentivizing High Quality Crowdwork,

C.-J. Ho, A. Slivkins, S. Suri, and J. W. V aughan, “Incentivizing High Quality Crowdwork,” in Proceedings of the 24th International Conference on World Wide Web , ser. WWW ’15. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2015, pp. 419–429, event-place: Florence, Italy. [Online]. Available: http...

arXiv 2015