A New Way to Look at Regional Survey Data: Differences in Vacancy Rates and Persons per Household by County, 2000-2005

Charles D. Coleman; Jonathan F. Takeuchi

A New Way to Look at Regional Survey Data: Differences in Vacancy Rates and Persons per Household by County, 2000-2005

T0 review · 2 major / 0 minor · reviewed 2026-05-24 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Maps of all 3,141 U.S. counties display survey estimates together with their significance levels to identify meaningful changes.

desk verdict This paper maps county-level differences in two variables between the 2000 Census and 2005 ACS but does not fix or even discuss the comparability problems between those sources. read the letter →

arxiv 2309.03002 v2 submitted 2023-09-06 stat.ME

Charles D. Coleman , Jonathan F. Takeuchi This is my paper

classification stat.ME

keywords regionalsurveydatacounty-levelmapsAmericanCommunityvacancyratespersonsperhouseholdstatisticalsignificancevisualizationCensus2000

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows how to map regional survey estimates and their significance levels at once across every U.S. county. This lets analysts spot statistically reliable differences between the 2000 Census and 2005 American Community Survey figures for vacancy rates and persons per household. The technique works for all but the smallest counties where uncertainty is too high. A reader would care because it turns large volumes of survey data into a practical visual tool for regional comparison.

What carries the argument

Dual-display maps that combine county-level estimates with significance levels or uncertainty measures.

What would settle it

Evidence that survey design changes between 2000 and 2005 introduced systematic bias not captured in the reported significance levels would show the maps do not reliably isolate meaningful differences.

Watch

Extended reading notes

Core claim

Regional survey estimates and their significance levels are simultaneously displayed in maps that show all 3,141 U.S. counties and equivalents, so an analyst can focus attention on significant differences or those with low-valued uncertainty measures for all but the very smallest counties, as illustrated with differences between Census 2000 and 2005 ACS values.

Load-bearing premise

The 2000 Census and 2005 ACS estimates are directly comparable in definition and coverage, and the significance levels accurately reflect sampling error without additional bias from survey design changes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core point is that the authors have produced maps covering all 3,141 US counties that show changes in vacancy rates and persons per household from 2000 to 2005, with some form of significance shading so users can zero in on the bigger shifts. That is the entire contribution. The technique itself is standard in spatial statistics, and the paper applies it to two existing public variables without new methods or derivations. On the plus side, the coverage is complete and the display goal is practical for anyone who already works with county ACS tables and wants a quick visual scan. If the maps are legible and the uncertainty layers are correctly rendered, an analyst could use them as a screening tool. The soft spots are more serious. The abstract supplies no equations, no description of how variances for the differences are obtained, and no mention of adjustments for the shift from full-enumeration Census to sample-based ACS. The 2005 ACS has a different frame, mode, weighting, and question wording; standard separate variance formulas do not automatically produce valid standard errors for the change, and the paper gives no evidence that it has handled this. The stress-test concern therefore stands: counties flagged as significant could reflect survey design artifacts rather than substantive movement. Without methods or validation, the central claim that analysts can safely focus on the highlighted counties does not hold up. This work is aimed at applied users of ACS data who need spatial summaries. It shows no engagement with the literature on cross-survey comparability, so it does not qualify as serious statistical thinking on its own terms. I would not bring it to a reading group, would not cite it, and would not send it for peer review.

Referee Report

2 major / 0 minor

Summary. The manuscript describes a visualization method for regional survey data in which maps simultaneously display county-level differences (vacancy rates and persons per household) between the 2000 Census and 2005 ACS for all 3,141 U.S. counties together with associated significance levels, enabling analysts to focus attention on statistically notable changes except in the smallest counties.

Significance. If the displayed significance levels correctly reflect sampling error in the differences and the two data sources are definitionally comparable, the maps would provide a practical tool for rapid identification of substantive regional change across a complete national set of small areas. The approach is primarily presentational rather than a methodological advance in estimation or inference.

major comments (2)

[Abstract] Abstract: the central claim that significance levels permit analysts to focus on real changes for nearly all counties rests on the unstated assumption that variances of the differences can be obtained by combining the separate survey variance formulas; no derivation or validation of the variance estimator for the cross-survey difference is supplied.
[Abstract] Abstract: the manuscript treats the 2000 Census (full enumeration) and 2005 ACS (sample survey with different frame, mode, weighting, and question wording) as directly comparable without bias correction or design-effect adjustment; this assumption is load-bearing for the interpretation of the highlighted 'significant' counties.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their comments on the manuscript. We respond to each major comment below and indicate planned revisions.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that significance levels permit analysts to focus on real changes for nearly all counties rests on the unstated assumption that variances of the differences can be obtained by combining the separate survey variance formulas; no derivation or validation of the variance estimator for the cross-survey difference is supplied.

Authors: The referee is correct that the manuscript provides no explicit derivation or validation of the variance for the difference. Because the 2000 Census is a full enumeration, its sampling variance is zero and the variance of the difference equals the ACS variance. We will revise the methods section to state this assumption of independence explicitly and include a short reference to the standard ACS variance formula. revision: yes
Referee: [Abstract] Abstract: the manuscript treats the 2000 Census (full enumeration) and 2005 ACS (sample survey with different frame, mode, weighting, and question wording) as directly comparable without bias correction or design-effect adjustment; this assumption is load-bearing for the interpretation of the highlighted 'significant' counties.

Authors: We agree that methodological differences between the sources exist and that the manuscript does not apply bias corrections. The contribution is presentational. In revision we will add a sentence to the abstract and a paragraph to the discussion noting these differences and directing readers to Census Bureau comparability documentation for the selected variables. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; purely descriptive visualization paper

full rationale

The manuscript describes a mapping technique to display county-level differences between 2000 Census and 2005 ACS estimates for vacancy rates and persons per household, together with significance levels. No equations, parameter fitting, predictions, uniqueness theorems, or derivation chains appear. The central claim is presentational: maps let analysts focus on statistically flagged differences. This does not reduce to any self-definition, fitted-input prediction, or self-citation load-bearing step. The comparability assumption noted by the skeptic is an external methodological concern, not a circularity in any claimed derivation.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review provides no technical content; no free parameters, axioms, or invented entities can be identified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A New Way to Look at Regional Survey Data: Differences in Vacancy Rates and Persons per Household by County, 2000-2005." pith.science (2026). https://pith.science/paper/2309.03002

@misc{pith2026230903002,
  author       = {Pith},
  title        = {Pith review of: A New Way to Look at Regional Survey Data: Differences in Vacancy Rates and Persons per Household by County, 2000-2005},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2309.03002}},
  note         = {Machine review of arXiv:2309.03002}
}

read the original abstract

Regional survey estimates and their significance levels are simultaneously displayed in maps that show all 3,141 U.S. counties and equivalents. An analyst can focus his attention on significant differences (or those with a different, low-valued uncertainty measure) for all but the very smallest counties. Differences between Census 2000 and the 2005 American Community Survey values are shown.

Figures

Figures reproduced from arXiv: 2309.03002 by the authors.

**Figure 2.** Differences in Persons Per Household: 2005 American Community Survey Less Census 2000 [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

**Figure 5.** p-Values of Differences in Vacancy Rates: 2005 ACS Estimates Less Census 2000 [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗

Figures from the paper (6 more)

**Figure 6.** Figure 6: p-Values of Differences in Persons Per Household: 2005 ACS Estimates Less Census 2000 [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Significant Differences in Vacancy Rates: 2005 ACS Estimates Less Census 2000. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

**Figure 8.** Figure 8: Significant Differences in Persons Per Household: 2005 ACS Estimates Less Census 2000. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

**Figure 9.** Figure 9: Significant Differences in Vacancy Rates: 2005 ACS Estimates Less Census 2000. [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]

**Figure 10.** Figure 10: Significant Differences in Persons Per Household: 2005 ACS Estimates Less Census 2000. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: Significant Differences in Vacancy Rates: 2005 ACS Estimates Less Census 2000. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

Mapping Census 2000: The Geography of U.S. Diversity,

Brewer, C. A., and Suchan, T.A. (2001), “Mapping Census 2000: The Geography of U.S. Diversity,” U.S. Census Bureau, Series CENSR01-1. Washington, DC: U.S. Government Printing Office. Francis, J., Toontisirin, N., Anantsuksomsri, S., Vink J., and Zheng V. (2015), “Alternative Strategies for Mapping ACS Estimates and Error of Estimation,” in Emerging Techni...

work page 2001
[2]

National Vacancy Rate and PPH Statistics. Vacancy Rate PPH ACS 2005 Estimate 0.109 2.511 Standard Error <0.001 0.001 Census 2000 Value 0.090 2.594 Difference, 2000 to 2005 0.010 -0.083 t (Difference) 277.12 56.09 p (two-sided) <.0001 <.0001 N 1902714 1768518 Sum of weights 122592161 109191370 Sources: U.S. Census Bureau, American Community Survey 2005, C...

work page 2005
[3]

p-values of County Vacancy Rate Differences. p-value Count Percent Expected Percent under G0 0.995+ 220 7.02 0.5 0.9750.995 201 6.41 2.0 0.950.975 162 5.17 2.5 Other 2370 75.62 90.0 0.0250.05 54 1.72 2.5 0.0050.025 49 1.56 2.0 <0.005 78 2.49 0.5 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: p-values for only 3,134 of 3...

work page 2005
[4]

p-values of County PPH Differences. p-value Count Percent Expected Percent under G0 0.995+ 0 0 0.5 0.9750.995 11 0.35 2.0 0.950.975 15 0.48 2.5 Other 2162 68.83 90.0 0.0250.05 230 7.32 2.5 0.0050.025 322 10.25 2.0 <0.005 401 12.77 0.5 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: All p-values are one-sided. 14 Table

work page 2005
[5]

Significance Level Count Percent 1% 298 9.51 5% 250 7.98 10% 216 6.89 Not significant 2370 75.62 Sources: U.S

Two-Sided Significance Levels for County Vacancy Rate Differences. Significance Level Count Percent 1% 298 9.51 5% 250 7.98 10% 216 6.89 Not significant 2370 75.62 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data. 15 Table

work page 2005
[6]

Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data

Two-Sided Significance Levels for County PPH Differences Significance Level Count Percent 1% 401 12.77 5% 333 10.60 10% 245 7.80 Not significant 2162 68.83 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data. 16 Table

work page 2005
[7]

Differences in Vacancy Rates: 2005 American Community Survey Less Census 2000 18 Figure

work page 2005
[8]

Differences in Persons Per Household: 2005 American Community Survey Less Census 2000 19 Figure

work page 2005

Show all 12 references

[9]

p-Values of Differences in Vacancy Rates: 2005 ACS Estimates Less Census

2005
[10]

p-Values of Differences in Persons Per Household: 2005 ACS Estimates Less Census

2005
[14]

Significant Differences in Persons Per Household: 2005 ACS Estimates Less Census

2005
[15]

Significant Differences in Vacancy Rates: 2005 ACS Estimates Less Census

2005

Pith tools

Reviewed May 24, 2026 · model on record in the stance chip above.

[1] [1]

Mapping Census 2000: The Geography of U.S. Diversity,

Brewer, C. A., and Suchan, T.A. (2001), “Mapping Census 2000: The Geography of U.S. Diversity,” U.S. Census Bureau, Series CENSR01-1. Washington, DC: U.S. Government Printing Office. Francis, J., Toontisirin, N., Anantsuksomsri, S., Vink J., and Zheng V. (2015), “Alternative Strategies for Mapping ACS Estimates and Error of Estimation,” in Emerging Techni...

work page 2001

[2] [2]

National Vacancy Rate and PPH Statistics. Vacancy Rate PPH ACS 2005 Estimate 0.109 2.511 Standard Error <0.001 0.001 Census 2000 Value 0.090 2.594 Difference, 2000 to 2005 0.010 -0.083 t (Difference) 277.12 56.09 p (two-sided) <.0001 <.0001 N 1902714 1768518 Sum of weights 122592161 109191370 Sources: U.S. Census Bureau, American Community Survey 2005, C...

work page 2005

[3] [3]

p-values of County Vacancy Rate Differences. p-value Count Percent Expected Percent under G0 0.995+ 220 7.02 0.5 0.9750.995 201 6.41 2.0 0.950.975 162 5.17 2.5 Other 2370 75.62 90.0 0.0250.05 54 1.72 2.5 0.0050.025 49 1.56 2.0 <0.005 78 2.49 0.5 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: p-values for only 3,134 of 3...

work page 2005

[4] [4]

p-values of County PPH Differences. p-value Count Percent Expected Percent under G0 0.995+ 0 0 0.5 0.9750.995 11 0.35 2.0 0.950.975 15 0.48 2.5 Other 2162 68.83 90.0 0.0250.05 230 7.32 2.5 0.0050.025 322 10.25 2.0 <0.005 401 12.77 0.5 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: All p-values are one-sided. 14 Table

work page 2005

[5] [5]

Significance Level Count Percent 1% 298 9.51 5% 250 7.98 10% 216 6.89 Not significant 2370 75.62 Sources: U.S

Two-Sided Significance Levels for County Vacancy Rate Differences. Significance Level Count Percent 1% 298 9.51 5% 250 7.98 10% 216 6.89 Not significant 2370 75.62 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data. 15 Table

work page 2005

[6] [6]

Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data

Two-Sided Significance Levels for County PPH Differences Significance Level Count Percent 1% 401 12.77 5% 333 10.60 10% 245 7.80 Not significant 2162 68.83 Sources: U.S. Census Bureau, American Community Survey 2005, Census 2000 Note: Significance levels shown are the minimum conventional significance levels of the data. 16 Table

work page 2005

[7] [7]

Differences in Vacancy Rates: 2005 American Community Survey Less Census 2000 18 Figure

work page 2005

[8] [8]

Differences in Persons Per Household: 2005 American Community Survey Less Census 2000 19 Figure

work page 2005