pith. sign in

arxiv: 2407.20240 · v3 · pith:BZULM7ZDnew · submitted 2024-07-15 · 💻 cs.CY · cs.AI

Social and Ethical Risks Posed by General-Purpose LLMs for Settling Newcomers in Canada

Isar Nejadgholi , Maryam Molamohammadi , Samir Bakhtawar This is my paper

Pith reviewed 2026-05-23 23:05 UTC · model grok-4.3

classification 💻 cs.CY cs.AI
keywords generative AILLMssettlement servicesCanadaimmigrationAI ethicsnewcomersrefugees
0
0 comments X

The pith

Ad-hoc use of general-purpose LLMs like ChatGPT can harm immigrants and refugees in Canada's settlement services.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how rising immigration pressures on Canada's non-profit settlement sector create demand for efficiency tools, which may lead newcomers and providers to turn to untailored generative AI. It argues that such general-purpose models are not designed for this domain and can produce detrimental outcomes for the people they are meant to help. The authors aim to warn against unguarded adoption while pushing for AI literacy efforts and customized systems that incorporate community input and human oversight. A sympathetic reader would see this as a practical question of whether off-the-shelf AI should shape life-changing integration processes without safeguards.

Core claim

The ad-hoc use of general-purpose generative AI such as ChatGPT might become common practice among newcomers and service providers, yet these tools are not tailored for the settlement domain and can have detrimental implications for immigrants and refugees.

What carries the argument

The mismatch between general-purpose LLMs and the specific operational and ethical requirements of the Canadian settlement sector, which lacks domain alignment and built-in human oversight.

If this is right

  • AI literacy programs become necessary for both newcomers and settlement providers.
  • Customized LLMs must be developed to align with the preferences of affected immigrant and refugee communities.
  • Any AI tools deployed in the sector require seamless integration with existing workflows plus explicit human oversight, trustworthiness, and accountability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same mismatch between general models and domain needs could appear in other public-service areas serving vulnerable groups, such as healthcare navigation or legal aid.
  • Field studies tracking actual LLM usage patterns in settlement offices could test whether harms materialize at scale.
  • Policy frameworks for AI in immigration services might need explicit requirements for community consultation before deployment.

Load-bearing premise

That general-purpose LLMs will be used ad-hoc in settlement work without tailored design or oversight and will produce detrimental effects.

What would settle it

Empirical data from settlement organizations showing that current ad-hoc LLM use produces no measurable negative impacts on newcomer outcomes or service quality.

Figures

Figures reproduced from arXiv: 2407.20240 by Isar Nejadgholi, Maryam Molamohammadi, Samir Bakhtawar.

Figure 5
Figure 5. Figure 5: Examples of images generated for refugee families (top row) and immigrant families (bot￾tom row). The two refugee families are described as Middle Eastern, and the immigrant families are described as being from South Asia (India or Pakistan) and practicing Hinduism, Islam, or Sikhism (left) and Middle Eastern and Muslim (right). Such an overgeneralized portrayal can also overshadow the rich and varied expe… view at source ↗
read the original abstract

The non-profit settlement sector in Canada supports newcomers in achieving successful integration. This sector faces increasing operational pressures amidst rising immigration targets, which highlights a need for enhanced efficiency and innovation, potentially through reliable AI solutions. The ad-hoc use of general-purpose generative AI, such as ChatGPT, might become a common practice among newcomers and service providers to address this need. However, these tools are not tailored for the settlement domain and can have detrimental implications for immigrants and refugees. We explore the risks that these tools might pose on newcomers to first, warn against the unguarded use of generative AI, and second, to incentivize further research and development in creating AI literacy programs as well as customized LLMs that are aligned with the preferences of the impacted communities. Crucially, such technologies should be designed to integrate seamlessly into the existing workflow of the settlement sector, ensuring human oversight, trustworthiness, and accountability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript argues that ad-hoc use of general-purpose LLMs such as ChatGPT by newcomers and settlement service providers in Canada risks detrimental effects due to lack of domain tailoring, hallucinations, bias, and absent provenance; it therefore issues a warning and calls for AI literacy programs plus development of customized, community-aligned LLMs that preserve human oversight, trustworthiness, and workflow integration.

Significance. If the forward-looking caution is borne out, the work draws attention to ethical deployment of generative AI in high-stakes administrative services for vulnerable populations, potentially shaping policy, literacy initiatives, and requirements for domain-specific models in the Canadian settlement sector.

major comments (1)
  1. [Abstract] Abstract: the central claim that general-purpose LLMs 'can have detrimental implications for immigrants and refugees' is asserted without concrete examples, documented incidents, or case studies from the settlement domain; this absence weakens the motivation for the recommended literacy programs and customized models.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the single major comment below and agree that the abstract can be strengthened to better motivate the paper's claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that general-purpose LLMs 'can have detrimental implications for immigrants and refugees' is asserted without concrete examples, documented incidents, or case studies from the settlement domain; this absence weakens the motivation for the recommended literacy programs and customized models.

    Authors: We agree that the abstract would benefit from brief, concrete illustrations drawn from the manuscript's risk analysis to strengthen motivation. The body of the paper examines domain-specific applications of well-documented LLM limitations (e.g., hallucinations producing incorrect guidance on immigration procedures or refugee claims processes; cultural biases that could disadvantage certain newcomer groups; and absent provenance for official settlement documents). In revision we will add one or two short, non-speculative examples to the abstract while preserving its concise form, thereby linking the general risks more explicitly to the settlement sector without requiring new empirical data. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a forward-looking ethics position piece whose central claim applies domain-general LLM properties (hallucination, bias, lack of provenance) to the Canadian settlement context as a cautionary argument; it contains no equations, derivations, fitted parameters, predictions, or self-citation chains that reduce the conclusion to its own inputs by construction. The argument is self-contained within its genre and does not rely on any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical or empirical content; the central claim rests on the untested premise that general-purpose tools will be misused in settlement contexts.

pith-pipeline@v0.9.0 · 5697 in / 967 out tokens · 15156 ms · 2026-05-23T23:05:48.354497+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 50 canonical work pages · 5 internal anchors

  1. [1]

    Government of Canada. 2023. Settlement Program. Retrieved from part-2-digital-trans- formation.pdf (canada.ca). Accessed: 2024-03-14

    work page 2023

  2. [2]

    Government of Canada. 2023. 2023 Settlemen t Outcomes Report, Data -driven Insights for the Future of the Settlement Program: Part 1 – Newcomer Outcomes Stock Take, Re- trieved from: part-2-digital-transformation.pdf (canada.ca), Accessed: 2024-03-14

    work page 2023

  3. [3]

    Government of Canada. 2023. Notice – Supplementary Information for the 2024 -2026 Immigration Levels Plan. Retrieved from: Notice – Supplementary Information for the 2024-2026 Immigration Levels Plan - Canada.ca, Accessed: 2024-03-14

    work page 2023

  4. [4]

    Government of Canada. 2023. 2023 Settleme nt Outcomes Report, Data -driven Insights for the Future of the Settlement Program: Part 2 – Digital Transformation, Retrieved from: part-2-digital-transformation.pdf (canada.ca), Accessed: 2024-03-14

    work page 2023

  5. [5]

    2023, Digital Border Governance: A Human Rights Based Approach, Digital Border Governance: A Human Rights Based Approach - Published September 2023 (ohchr.org)

    Office of the United Nations High Commissioner for Human Rights. 2023, Digital Border Governance: A Human Rights Based Approach, Digital Border Governance: A Human Rights Based Approach - Published September 2023 (ohchr.org). Accessed: 2024-03-14

    work page 2023

  6. [6]

    Asheesh Moosapeta. 2023. How you can use ChatGPT on your immigration journey. CIC NEWS. How you can use ChatGPT on your immigration journey | CIC News. Accessed: 2024-03-14

    work page 2023

  7. [7]

    Petra Molnar. 2023. Building knowledge about generative AI with mobile population s. Building Knowledge about Generative AI with Mobile Populations | by Petra Molnar | Berkman Klein Center Collection | Medium . Accessed: 2024-03-14

    work page 2023

  8. [8]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mish- kin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural infor- mation processing systems. 35 (2022), 27730–27744

    work page 2022

  9. [9]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems. 36 (2024)

    work page 2024

  10. [10]

    Jing Xu, Da Ju, Margaret Li, Y -Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079 (2020)

    work page arXiv 2020

  11. [11]

    Yuntao Ba i, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das- Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  12. [12]

    Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of “Bias” in NLP . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5454–5476

    work page 2020

  13. [13]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  14. [14]

    Leonardo Nicoletti and Dina Bas. 2023. Humans are Biased. Generative AI is even worse. Bloomberg Technology +Equity. Retrieved from Generative AI Takes Stereotypes and Bias From Bad to Worse (bloomberg.com). Accessed: 2024-03-14

    work page 2023

  15. [15]

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng

    work page

  16. [16]

    In Findings of the Association for Computational Linguistics: EMNLP 2023, 3730–3748

    Kelly is a Warm Person, Joseph is a Role Model: Gender Biases in LLM-Generated Reference Letters. In Findings of the Association for Computational Linguistics: EMNLP 2023, 3730–3748. 25

    work page 2023

  17. [17]

    Paresh Dave. 2023. ChatGPT Is Cutting Non-English Languages Out of the AI Revolu- tion. Retrieved from ChatGPT Is Cutting Non-English Languages Out of the AI Revolu- tion | WIRED. Accessed: 2024-03-14

    work page 2023

  18. [18]

    Regina Ta and Nicol Turner Lee. 2023. How language gaps constrain generative AI development. Retrieved from How language gaps constrain generative AI development | Brookings. Accessed: 2024-03-14

    work page 2023

  19. [19]

    Shirin Ghaffary. 2019. The algorithms that detect hate speech online are biased against black people. The algorithms that detect hate speech online are biased against black peo- ple - Vox, Accessed: 2024-03-14

    work page 2019

  20. [20]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Lan- guage Generation. ACM Comput. Surv. 55, 12, Article 248, 38 pages

    work page 2023

  21. [21]

    Muru Zhang, Ofir Press, William Merrill, Alisa Li u, and Noah A Smith. 2023. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534 (2023)

    work page arXiv 2023

  22. [22]

    Trevor Deley and Elizabeth Dubois. 2020. Assessing trust versus reliance for technology platforms by systematic literature review. Social Media+ Society 6, 2 (2020)

    work page 2020

  23. [23]

    Nenad Tomasev, Kevin R McKee, Jackie Kay, and Shakir Mohamed. 2021. Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 254–265

    work page 2021

  24. [24]

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19). 267–284

    work page 2019

  25. [25]

    David Garcia, Mansi Goel, Amod Kant Agrawal, and Ponnurangam Kumaraguru. 2018. Collective aspects of privacy in the Twitter social network. EPJ Data Science 7, 1 (2018), 1–13

    work page 2018

  26. [26]

    John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvort- sova, Johann George, Natalija Gucevska, Mark Harman, Ralf Lämmel, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers. 2020. WES: Agent-based User Interaction Simulation on Real Infrastructure. In Proceedings of the IEEE/ACM 42nd International Conference on Software Enginee...

    work page 2020

  27. [27]

    Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Ta...

    work page 2022

  28. [28]

    Canada Newswire. 2023. Interac survey shows more than half of new Canadian families have been targeted by financial fraud. Retrieved from Interac survey shows more than half of new Canadian families have been targeted by financial fraud - Interac, Accessed: 2024-03-14

    work page 2023

  29. [29]

    Luciano Floridi, Josh Cowls, Thomas C King, and Mariarosaria Taddeo. 2021. How to design AI for social good: Seven essential factors. Ethics, Governance, and Policies in Artificial Intelligence (2021), 125–151

    work page 2021

  30. [30]

    Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738 (2023)

    work page arXiv 2023

  31. [31]

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Ariana Martino, Michael Iannelli, an d Coleen Truong. 2023. Knowledge injection to counter large language model (LLM) hallucination. In European Semantic Web Confer- ence. Springer, 182–185. 26

    work page 2023

  33. [33]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Yaqi Xie, Ziwei Xu, Mohan S Kankanhalli, Kuldeep S Meel, and Haro ld Soh. 2019. Embedding symbolic knowledge into deep networks. Advances in neural information processing systems 32 (2019)

    work page 2019

  35. [35]

    Ziwei Xu, Yogesh Rawat, Yongkang Wong, Mohan S Kankanhalli, and Mubarak Shah

    work page

  36. [36]

    Advances in Neural Information Processing Systems 35 (2022), 14890 – 14903

    Don’t Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation. Advances in Neural Information Processing Systems 35 (2022), 14890 – 14903

    work page 2022

  37. [37]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al . 2023. Survey on fac- tuality in large language models: Knowledge, retrieval and domain-specificity. arXiv pre- print arXiv preprint arXiv:2310.07521 (2023)

    work page arXiv 2023

  38. [38]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Ce- likyilmaz, and Jason Weston. 2023. Chain -of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  39. [39]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems 35 (2022), 24824 – 24837

    work page 2022

  40. [40]

    Shunyu Yao, Dian Yu, Jeffre y Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem-solving with large lan- guage models. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  41. [41]

    Nikita Mehandru, Sweta Agrawal, Yimin X iao, Ge Gao, Elaine Khoong, Marine Carpuat, and Niloufar Salehi. 2023. Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 11633–11647

    work page 2023

  42. [42]

    Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Ap- plications with Programmable Rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 431–445

    work page 2023

  43. [43]

    Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. The Partici- patory Turn in AI Design: Theoretical Foundations and the Current State of Practice. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mecha- nisms, and Optimization (EAAMO '23) . Association for Co mputing Machinery, New York, NY, USA, Article 37, 1–23

    work page 2023

  44. [44]

    Elizabeth Bondi, Lily Xu, Diana Acosta-Navas, and Jackson A. Killian. 2021. Envision- ing Communities: A Participatory Approach Towards AI for Social Good. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES '21). Association for Computing Machinery, New York, NY, USA, 425–436

    work page 2021

  45. [45]

    Sasha Costanza -Chock. 2020. Design justice: Community -led practices to build the world we need. The MIT Press

    work page 2020

  46. [46]

    Valerio Capraro , Austin Lentsch, Daron Acemoglu, Selin Akgun, Aisel Akhmedova, Ennio Bilancini, Jean -François Bonnefon, Pablo Brañas -Garza, Luigi Butera, Karen M Douglas, et al . 2024. The impact of generative artificial intelligence on socioeconomic inequalities and policy making. PNAS nexus 3, 6 (2024)

    work page 2024

  47. [47]

    Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic lies: Understanding AI-generated misinformation and 27 evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–20

    work page 2023

  48. [48]

    Government of Canada. 2021. 2021 Settlement Outcomes Highlights Report Summary Findings from IRCC’s First Settlement Outcomes Report 2021, Retrieved from: Settle- ment Outcomes HIGHLIGHTS Report (canada.ca), Accessed: 2024-03-14

    work page 2021

  49. [49]

    Lucia Nalbandian. 2022. An eye for an ‘I:’a critical assessment of artificial intelligence tools in migration and asylum management. Comparative Migration Studies 10, 1 (2022), 32

    work page 2022

  50. [50]

    Mayowa Oluwasanmi. 2024. The Rise of AI and Technology in Immigration Enforce- ment. The Regulatory Review, March 23, 2024. Retrieved from The Rise of AI and Tech- nology in Immigration Enforcement | The Regulatory Review (theregreview.org)

    work page 2024