pith. sign in

arxiv: 2604.12324 · v1 · submitted 2026-04-14 · 📊 stat.AP · cs.CY

HICM: An approach towards Harmonizing Indian Census Migration data and its applications

Nivedita Batra , Chiranjoy Chattopadhyay , Mayurakshi Chaudhuri This is my paper

Pith reviewed 2026-05-10 14:39 UTC · model grok-4.3

classification 📊 stat.AP cs.CY
keywords Indian censusmigration datadata harmonizationbias correctioninterstate migrationtemporal analysisdata imputationrepresentativeness bias
0
0 comments X

The pith

The HICM framework corrects measurement and representativeness biases in Indian census migration data to produce more consistent structures for temporal analysis.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Indian migration data from decennial censuses suffer from systematic gaps due to uneven coverage and measurement inconsistencies across states and time periods. The paper introduces the HICM framework to identify and mitigate two main biases through pre-processing, mitigation, and validation steps based on statistical diagnostics. An empirical evaluation demonstrates that this bias-aware correction leads to substantially improved consistency in the data structure and more reliable results in subsequent temporal analyses. This approach supplies reproducible imputation and smoothing methods that create a stronger base for longitudinal studies of internal migration flows.

Core claim

Empirical evaluation of the harmonized interstate migration data shows that bias-aware correction through the HICM framework substantially improves consistency in the data structure and enhances the reliability of temporal analysis results, advancing migration analytics via reproducible data imputation and smoothing for policy-relevant longitudinal network analysis.

What carries the argument

HICM framework, which applies principled pre-processing, mitigation, and validation strategies grounded in statistical diagnostics to address measurement bias and representativeness bias.

Load-bearing premise

The identified measurement and representativeness biases can be effectively mitigated through the proposed pre-processing, mitigation, and validation strategies without introducing new biases or losing critical information from the original data.

What would settle it

Re-running the temporal analyses on the harmonized data and finding no reduction in inconsistencies in migration flow estimates compared to the raw census records would falsify the improvement claim.

Figures

Figures reproduced from arXiv: 2604.12324 by Chiranjoy Chattopadhyay, Mayurakshi Chaudhuri, Nivedita Batra.

Figure 1
Figure 1. Figure 1: Flow diagram summarizing the process of applying harmonization strategies on various bias profiles. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of 1991 migrant inflow population: Ground truth vs Imputed data [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Violin plot for decade-wise comparison of migrant inflow after imputing missing states data in the 1991 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Violin plots for comparison of 2011 data before and after redistributing “unclassifiable” migrants. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Communities of migrants evolving over three decades (based on total interstate migrant stocks). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Communities of migrants in the 1991 dataset before and after imputation [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
read the original abstract

Reliable analysis of migration is critically dependent on the quality and consistency of the underlying data. Indian migration data, primarily derived from decennial census records, are affected by systematic gaps arising from uneven coverage and measurement inconsistencies across states and time. This paper presents a data-centric framework, HICM, for harmonizing Indian census migration data recorded under the Indian census and correcting prominent sources of bias prior to downstream analyses. We explicitly identify two types of bias across three decades of migration data: measurement bias and representativeness bias. We propose to address these gaps through principled pre-processing, mitigation, and validation strategies grounded in statistical diagnostics. An empirical evaluation of harmonized Indian interstate migration data reveals that bias-aware data correction substantially improves the consistency in the structure of the data and enhances the reliability of subsequent temporal analysis results. By improving data quality through reproducible data imputation and smoothing, this work advances migration analytics and provides a robust foundation for policy-relevant longitudinal network analysis of Indian internal migration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents HICM, a data-centric framework for harmonizing Indian census migration data by identifying and correcting measurement bias and representativeness bias across three decades using pre-processing, mitigation, and statistical-diagnostic validation strategies. The key claim is that an empirical evaluation of the harmonized interstate migration data shows that this bias-aware correction substantially improves data structure consistency and enhances the reliability of subsequent temporal analyses, providing a foundation for policy-relevant longitudinal network analysis.

Significance. If the quantitative improvements are rigorously demonstrated, this framework would offer a practical, reproducible method for correcting biases in Indian census migration data, supporting more reliable longitudinal network analyses and policy insights in demography and applied statistics. The emphasis on statistical diagnostics and data imputation is a strength for reproducibility.

major comments (2)
  1. Abstract: the central claim that bias-aware correction 'substantially improves the consistency in the structure of the data' is unsupported by any reported quantitative metrics, before/after comparisons, statistical tests, or error bars from the empirical evaluation, which is load-bearing for the paper's main result.
  2. Methods/Validation section: the specific statistical diagnostics for identifying measurement and representativeness biases, along with the exact pre-processing and mitigation steps, are not described in sufficient detail to evaluate whether they avoid introducing new biases or losing critical information, as required to substantiate the weakest assumption.
minor comments (2)
  1. Abstract: the acronym HICM is not expanded, which reduces clarity for readers.
  2. Abstract: the phrase 'three decades of migration data' should specify the exact census years covered to provide necessary context.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our manuscript. The comments highlight important areas where additional clarity and evidence will strengthen the presentation of the HICM framework. We address each major comment below and will incorporate revisions to improve the rigor and reproducibility of the work.

read point-by-point responses

  1. Referee: Abstract: the central claim that bias-aware correction 'substantially improves the consistency in the structure of the data' is unsupported by any reported quantitative metrics, before/after comparisons, statistical tests, or error bars from the empirical evaluation, which is load-bearing for the paper's main result.

    Authors: We agree that the abstract claim requires explicit quantitative support to be fully substantiated. In the revised manuscript, we will add specific metrics to the abstract and results, including before/after comparisons of network density, edge correlation coefficients across census rounds, and statistical tests (e.g., Mantel tests or chi-square goodness-of-fit for structural consistency) with associated p-values and error bars derived from bootstrap resampling. These will be tied directly to the empirical evaluation section. revision: yes

  2. Referee: Methods/Validation section: the specific statistical diagnostics for identifying measurement and representativeness biases, along with the exact pre-processing and mitigation steps, are not described in sufficient detail to evaluate whether they avoid introducing new biases or losing critical information, as required to substantiate the weakest assumption.

    Authors: We acknowledge the need for greater methodological transparency. The revised version will expand the Methods and Validation sections with precise descriptions of the diagnostics (including formulas for bias detection thresholds and representativeness checks), step-by-step pre-processing procedures (e.g., exact imputation algorithms and smoothing parameters), and mitigation techniques. We will also include a new subsection on sensitivity analyses demonstrating that the corrections do not introduce artifacts or discard essential migration signals. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper describes a data harmonization framework (HICM) that identifies measurement and representativeness biases in Indian census migration records, then applies pre-processing, mitigation, and statistical-diagnostic validation steps. No equations, fitted parameters, or predictions are presented that reduce by construction to the paper's own inputs. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. The central claim—that bias-aware correction improves data consistency—is evaluated empirically against external data properties and is therefore falsifiable outside any internal fit. The work is self-contained as a reproducible data-processing pipeline grounded in standard statistical diagnostics.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are explicitly mentioned or identifiable from the abstract. The framework relies on unspecified statistical diagnostics and pre-processing whose detailed assumptions cannot be audited here.

pith-pipeline@v0.9.0 · 5476 in / 1263 out tokens · 41542 ms · 2026-05-10T14:39:29.190350+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    When census is an election: A game-theoretic analysis of over-reporting of headcount

    Vikas Kumar. When census is an election: A game-theoretic analysis of over-reporting of headcount. InPower and Responsibility: Interdisciplinary Perspectives for the 21st Century in Honor of Manfred J. Holler, pages 373–393. Springer, 2023

    work page 2023

  2. [2]

    Census laws and the quality of census data: The limits of punitive legislation.Statistical Journal of the IAOS, 36(4):1143–1160, 2020

    Vikas Kumar. Census laws and the quality of census data: The limits of punitive legislation.Statistical Journal of the IAOS, 36(4):1143–1160, 2020

    work page 2020

  3. [3]

    ‘census towns’ in india and what it means to be ‘urban’: Competing epistemologies and potential new approaches.Singapore Journal of Tropical Geography, 38(2):229–244, 2017

    Srilata Sircar. ‘census towns’ in india and what it means to be ‘urban’: Competing epistemologies and potential new approaches.Singapore Journal of Tropical Geography, 38(2):229–244, 2017

    work page 2017

  4. [4]

    The quality of age data: Comparison between two recent indian censuses 2001–2011.Clinical Epidemiology and Global Health, 8(2):371–376, 2020

    Akhilesh Yadav, Minakshi Vishwakarma, and Shekhar Chauhan. The quality of age data: Comparison between two recent indian censuses 2001–2011.Clinical Epidemiology and Global Health, 8(2):371–376, 2020

    work page 2001

  5. [5]

    Age at marriage of indian women

    Aparajita Chattopadhyay and Akancha Singh. Age at marriage of indian women. InAtlas of Gender and Health Inequalities in India, pages 115–123. Springer, 2024

    work page 2024

  6. [6]

    Counting power: The caste census and the four faces of exclusion in india

    Yasha Singh and Nitin Ranjan. Counting power: The caste census and the four faces of exclusion in india. 2025

    work page 2025

  7. [7]

    Caste census data for a just republic

    Yasha Singh and Nitin Ranjan. Caste census data for a just republic. 2025

    work page 2025

  8. [8]

    Caste census: A detailed report

    Nitin Ranjan and Yasha Singh. Caste census: A detailed report. 2025

    work page 2025

  9. [9]

    Measuring caste in India.Pew Research Center (Decoded), June, 29, 2021

    Kelsey Jo Starr and Neha Sahgal. Measuring caste in India.Pew Research Center (Decoded), June, 29, 2021

    work page 2021

  10. [10]

    Jayanta Datta and Prasenjit Sinha. Spatial patterns of heaping in age data among literates, illiterates, and numeracy– literacy correlates: A cross-sectional analysis of census 2011, of india.Indian Journal of Community Medicine, 49(1):189–194, 2024

    work page 2011

  11. [11]

    Indian census data on migration.The Oriental Anthropologist, 13(1):17–22, 2013

    Pranati Datta. Indian census data on migration.The Oriental Anthropologist, 13(1):17–22, 2013

    work page 2013

  12. [12]

    Nature of migration and its contribution to india’s urbanisation

    RB Bhagat. Nature of migration and its contribution to india’s urbanisation. InInternal Migration in Contemporary India, pages 23–40. Routledge India, 2025

    work page 2025

  13. [13]

    A computational study of indian interstate migration through the gender lens

    Niveditta Batra, Mayurakshi Chaudhuri, and Chiranjoy Chattopadhyay. A computational study of indian interstate migration through the gender lens. InIndia Migration Report 2023, pages 314–335. Routledge India, 2024

    work page 2023

  14. [14]

    An enquiry into the nature and problems of migration in india: A critical look

    Mahibul Islam and Subrata Saha. An enquiry into the nature and problems of migration in india: A critical look. International Research Journal of Economics and Management Studies IRJEMS, 3(3), 2024

    work page 2024

  15. [15]

    Indian diaspora population and space: national register, un global migration database and big data.Diaspora Studies, 12(2):134–159, 2019

    Nachatter Singh Garha and Andreu Domingo. Indian diaspora population and space: national register, un global migration database and big data.Diaspora Studies, 12(2):134–159, 2019

    work page 2019

  16. [16]

    Internal migration in india.Human Development in India, 2011

    Ravi Srivastava. Internal migration in india.Human Development in India, 2011

    work page 2011

  17. [17]

    Impact of internal migration in india.Refugee and Migratory Movements Research Unit (RMMRU)

    Ravi Srivastava. Impact of internal migration in india.Refugee and Migratory Movements Research Unit (RMMRU). http://www. rmmru. org/newsite/wp-content/uploads/2013/08/workingpaper41. pdf Social Security for Internal Migrants, 237, 2013

    work page 2013

  18. [18]

    Trends in mobility in india: issues of labour market integration and exclusion of vulnerable sections of the population.Area Development and Policy, 4(4):346–366, 2019

    Amitabh Kundu. Trends in mobility in india: issues of labour market integration and exclusion of vulnerable sections of the population.Area Development and Policy, 4(4):346–366, 2019

    work page 2019

  19. [19]

    Cambridge University Press, 2024

    Vikas Kumar.Numbers as political allies: The census in Jammu and Kashmir. Cambridge University Press, 2024

    work page 2024

  20. [20]

    Migration persistence across twentieth century india.Migration and Development, 1(1):87–112, 2012

    Chinmay Tumbe. Migration persistence across twentieth century india.Migration and Development, 1(1):87–112, 2012

    work page 2012

  21. [21]

    Is nativity on rise? estimation of interstate migration based on census of india 2011 for major states in india.Social Change, 45(1):137–144, 2015

    Bhaswati Das and Avijit Mistri. Is nativity on rise? estimation of interstate migration based on census of india 2011 for major states in india.Social Change, 45(1):137–144, 2015

    work page 2011

  22. [22]

    Social networks and internal migration: Evidence from facebook in india

    Harshil Sahai and Michael Bailey. Social networks and internal migration: Evidence from facebook in india. Technical report, Working Paper, 2023

    work page 2023

  23. [23]

    Short-term labor migration from rural north india: Evidence from new survey data.Population Research and Policy Review, 34(3):361–380, 2015

    Diane Coffey, John Papp, and Dean Spears. Short-term labor migration from rural north india: Evidence from new survey data.Population Research and Policy Review, 34(3):361–380, 2015

    work page 2015

  24. [24]

    Short-duration migration in india

    Vijay Korra. Short-duration migration in india. InIndia Migration Report 2011, pages 52–71. Routledge India, 2012

    work page 2011

  25. [25]

    Study of changing trends and patterns of internal migration and factors affecting it.International Journal, 1(4), 2020

    Anushka Gupta and Mehak Miglani. Study of changing trends and patterns of internal migration and factors affecting it.International Journal, 1(4), 2020

    work page 2020

  26. [26]

    Internal labour migration in india: Emerging needs of comprehensive national migration policy

    Anjali Borhade. Internal labour migration in india: Emerging needs of comprehensive national migration policy. InInternal migration in contemporary India, pages 212–252. Routledge India, 2025. 10

    work page 2025