Automated Profile Inference with Language Model Agents

Bolin Ding; Hanshen Xiao; Jingren Zhou; Ninghui Li; Yaliang Li; Yuntao Du; Zitao Li

arxiv: 2505.12402 · v2 · submitted 2025-05-18 · 💻 cs.CR

Automated Profile Inference with Language Model Agents

Yuntao Du , Zitao Li , Bolin Ding , Yaliang Li , Hanshen Xiao , Jingren Zhou , Ninghui Li This is my paper

Pith reviewed 2026-05-22 14:24 UTC · model grok-4.3

classification 💻 cs.CR

keywords automated profile inferencelanguage model agentsprivacy riskspseudonymous platformspersonal attribute extractionLLM agentsonline privacyAutoProfiler

0 comments

The pith

Language model agents can automatically extract sensitive personal attributes from public activities on pseudonymous platforms.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper examines how large language model agents can be used to automatically infer personal information from users' public posts on sites where people use pseudonyms. The authors present AutoProfiler, a system with four specialized agents that gather activities, process them, and build profiles containing identifiable and sensitive details. Tests on real-world and synthetic data show the system works well and efficiently, indicating that such inferences pose real privacy risks. A reader would care because it shows how AI collaboration can turn public data into private insights, challenging the safety of online anonymity.

Core claim

The central discovery is that an automated profiling framework called AutoProfiler, consisting of four LLM agents working collaboratively, can retrieve and process user online activities on pseudonymous platforms to generate profiles with extracted personal information. Experimental results demonstrate that AutoProfiler is highly effective and efficient, and that the inferred attributes are both identifiable and sensitive, thereby posing significant privacy risks.

What carries the argument

AutoProfiler, a collaborative framework of four specialized LLM agents that retrieve, process, and infer personal profiles from public user activities.

If this is right

Automated LLM agents can breach the protection of pseudonymity by linking public activities to personal attributes.
The inferred profiles contain information that is both identifiable and sensitive.
Such attacks can be carried out efficiently in real-world scenarios using existing datasets.
Exploration of mitigation strategies from various perspectives becomes necessary to address this threat.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Users on pseudonymous platforms may need to limit the details in their public posts to reduce inference risks.
This approach could extend to other AI models, suggesting broader implications for automated privacy attacks.
Platforms might consider technical measures to obscure user activity patterns that enable such inferences.

Load-bearing premise

Public user activities on pseudonymous platforms contain sufficient extractable information for LLM agents to infer personal attributes accurately without major errors or hallucinations.

What would settle it

A test where AutoProfiler is applied to a controlled set of user activities with known ground-truth attributes, and the accuracy of inferred attributes is measured to see if it exceeds random chance or shows frequent hallucinations.

Figures

Figures reproduced from arXiv: 2505.12402 by Bolin Ding, Hanshen Xiao, Jingren Zhou, Ninghui Li, Yaliang Li, Yuntao Du, Zitao Li.

**Figure 2.** Figure 2: Illustration of the key profiling processes in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Analysis of the categorized attributes of Reddit users by category, count, and estimated privacy risks. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Inferring attributes on Reddit. Sensitive in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Profiling from anonymized Tweets. “Invisible [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 7.** Figure 7: Verified users used in the Twitter dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 6.** Figure 6: SubReddit lists used in the Reddit dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

**Figure 8.** Figure 8: Word cloud of inferred personal attributes. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: Assigned sensitivity and identifiability scores [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

**Figure 10.** Figure 10: Calibration accuracy (Cal-Acc) of AutoProfiler on the SynthPAI dataset. Hardness and certainty scores are labeled by humans, and confidence is generated by AutoProfiler during prediction. Higher scores indicate greater difficulty, certainty, or confidence, respectively. The results indicate the inferences made by humans and AutoProfiler are generally well-aligned [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗

**Figure 13.** Figure 13: Complete prompts used for the Retriever agent. The tool instructions are auto-generated by AgentScope (Gao et al., 2024) and are omitted here. ## What Is the Task: You are an information summarizer who has ten years experiences of analyzing, organizing and outlining personal information. Your task is to analyze, check the correctness, and improve the reasonableness of a collection of user attributes infer… view at source ↗

**Figure 11.** Figure 11: Complete prompts used for the Strategist [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗

**Figure 14.** Figure 14: Prompts used for the Summarizer agent. As indicated, GPT-4 performs well in categorizing “Identifier” and “Geographic” information but exhibits slightly higher error rates in sensitive categories, such as “Finance” and “Behavior”. F.3 Prompts for Evaluation on Twitter The prompts for attribute-level and identity-level evaluation are shown in [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

**Figure 12.** Figure 12: Complete prompts used for the Extractor agent. F Complete Prompts F.1 Prompts for AutoProfiler Agents Figures 11, 12, 13, and 14 present the complete set of prompts used for Strategist, Retriever, Extractor and Summarizer, respectively. These prompts define the roles of agents in conducting automated profiling tasks. Specifically, each prompt outlines the agents’ responsibilities and requires them to pro… view at source ↗

**Figure 15.** Figure 15: Complete prompts used for categorizing inferred attributes. Attribute Inference Attack. The goal of an attribute inference attack is to infer sensitive attributes of target users or records using auxiliary information. Prior studies (Zheleva and Getoor, 2009; Kosinski et al., 2013; Gong and Liu, 2018) have shown that online behaviors, such as Facebook likes, can be exploited to infer sensitive attribut… view at source ↗

read the original abstract

Impressive progress has been made in automated problem-solving by the collaboration of large language model (LLM) based agents. However, these automated capabilities also open avenues for malicious applications. In this paper, we study a new threat that LLMs pose to online pseudonymity, called automated profile inference, where an adversary can instruct LLMs to automatically collect and extract sensitive personal attributes from publicly available user activities on pseudonymous platforms. We also introduce an automated profiling framework called AutoProfiler to demonstrate and assess the feasibility of such attacks in real-world scenarios. AutoProfiler consists of four specialized LLM agents that work collaboratively to retrieve and process user online activities and generate a profile with extracted personal information. Experimental results on two real-world datasets and one synthetic dataset show that AutoProfiler is highly effective and efficient, and the inferred attributes are both identifiable and sensitive, posing significant privacy risks. We explore mitigation strategies from different perspectives and advocate for increased public awareness of this emerging privacy threat.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

AutoProfiler frames automated multi-agent LLM profiling as a fresh threat to pseudonymity and shows it can pull attributes from public traces, but the results hinge on internal consensus instead of external ground truth.

read the letter

The main point is that the authors built a four-agent system called AutoProfiler that splits up the work of fetching posts, extracting details, and building profiles from pseudonymous activity. They test it on two real datasets plus a synthetic one and report that it surfaces identifiable and sensitive attributes efficiently. That setup is the concrete new piece: moving from manual or single-LLM profiling to an automated collaborative loop that handles retrieval and inference together. They also sketch some mitigation angles like platform changes and user awareness, which keeps the paper from being purely alarmist. The threat model itself is timely given how much public data sits on forums and social sites. The experiments give a sense of scale and speed that earlier manual studies lacked. The soft spot is exactly what the stress test flags. Success appears measured by whether the agents agree or by some sensitivity score rather than by checking the inferred attributes against known facts about the same users. Without that external anchor, high accuracy numbers could reflect plausible-sounding outputs more than real fidelity, especially with LLMs prone to confident errors. If the full paper does not add cross-linked or self-reported labels for verification, the privacy-risk conclusion rests on weaker footing than the abstract suggests. This is for the privacy and security crowd who track LLM misuse. Anyone working on anonymity tools or platform design would find the agent architecture and the concrete examples useful to think about. It is coherent enough and grounded in a real problem that it should go to referees rather than get desk-rejected. I would send it for review but flag the need for stronger validation experiments in the first round.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces AutoProfiler, a collaborative framework of four specialized LLM agents designed to automatically collect and extract sensitive personal attributes from publicly available user activities on pseudonymous platforms. It presents this as a new threat to online pseudonymity and evaluates the framework through experiments on two real-world datasets and one synthetic dataset, claiming high effectiveness and efficiency with identifiable and sensitive inferences that pose significant privacy risks. Mitigation strategies are also discussed.

Significance. If the central claims hold under rigorous validation, the work offers a timely empirical demonstration of an LLM-agent-based attack framework on pseudonymity. It contributes to privacy research by illustrating how public activity traces can be systematically processed into personal profiles, which could inform platform design, user awareness, and defenses against automated inference attacks.

major comments (2)

[Section 4] Section 4: The reported effectiveness of AutoProfiler is assessed via LLM-agent consensus or sensitivity scoring rather than independent ground-truth validation (e.g., self-reported attributes or cross-linked data). Without external verification, the results risk overstatement from hallucinations or plausible-sounding errors, which is load-bearing for the conclusion that the inferred attributes constitute a practical privacy threat.
[Experimental results] Experimental results: The manuscript provides no details on baselines, error bars, data selection criteria, or controls for LLM variability (e.g., temperature, model version). This absence makes the strong claims of high effectiveness difficult to interpret or reproduce, directly affecting confidence in the privacy-risk assessment.

minor comments (2)

[Framework description] The roles and interaction protocol of the four LLM agents could be described with greater precision, including any specific prompting strategies or handoff mechanisms between agents.
[Figures and tables] Figure captions and table headers would benefit from explicit definitions of all metrics used to quantify 'identifiable' and 'sensitive' attributes.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. We have reviewed the major comments carefully and provide point-by-point responses below, indicating where revisions will be made to improve the rigor and clarity of our evaluation of AutoProfiler.

read point-by-point responses

Referee: [Section 4] Section 4: The reported effectiveness of AutoProfiler is assessed via LLM-agent consensus or sensitivity scoring rather than independent ground-truth validation (e.g., self-reported attributes or cross-linked data). Without external verification, the results risk overstatement from hallucinations or plausible-sounding errors, which is load-bearing for the conclusion that the inferred attributes constitute a practical privacy threat.

Authors: We acknowledge the referee's concern regarding the absence of independent ground-truth validation. The pseudonymous nature of the platforms in our real-world datasets makes obtaining self-reported attributes or cross-linked data ethically and practically challenging, as it would require direct user contact or de-anonymization that could introduce new privacy risks. Our current evaluation relies on multi-agent consensus and sensitivity scoring to assess inference quality, with manual inspection on sampled outputs to check for plausibility. To address this, we will revise Section 4 to explicitly discuss these methodological limitations, including risks of hallucinations, and add results from the synthetic dataset (where ground truth is known by construction) as a complementary validation. We will also include a clearer description of how agent collaboration helps mitigate individual errors. revision: partial
Referee: [Experimental results] Experimental results: The manuscript provides no details on baselines, error bars, data selection criteria, or controls for LLM variability (e.g., temperature, model version). This absence makes the strong claims of high effectiveness difficult to interpret or reproduce, directly affecting confidence in the privacy-risk assessment.

Authors: We agree that additional experimental details are essential for reproducibility and proper interpretation of our results. In the revised manuscript, we will expand the experimental section to specify: the data selection criteria used for the two real-world datasets and the synthetic dataset; any baselines or comparison approaches employed; error bars or variance measures obtained from multiple runs; and controls for LLM variability, including the use of temperature=0 for deterministic behavior and the exact model versions and prompting parameters. These additions will strengthen the presentation of our effectiveness claims. revision: yes

Circularity Check

0 steps flagged

Empirical framework demonstration with no derivation chain

full rationale

The paper introduces AutoProfiler as an LLM-agent framework for automated profile inference and evaluates its effectiveness through experiments on two real-world datasets and one synthetic dataset. No mathematical derivations, first-principles predictions, or equations are claimed; results are presented as direct empirical outcomes of the agent collaboration rather than reductions to fitted parameters or self-referential inputs. Any self-citations serve only as background and do not load-bear the central feasibility demonstration. The work is therefore self-contained as an attack feasibility study without circular structure.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim depends on the assumption that LLM agents can reliably process natural language from user activities to extract accurate personal attributes, plus the existence of sufficient public data containing such attributes.

axioms (1)

domain assumption LLM agents can collaborate effectively to retrieve, process, and extract information from unstructured online text.
Invoked in the description of the AutoProfiler framework consisting of four specialized agents.

invented entities (1)

AutoProfiler no independent evidence
purpose: Automated profiling framework using four LLM agents to demonstrate the attack feasibility.
New system introduced to show the threat in real-world scenarios.

pith-pipeline@v0.9.0 · 5706 in / 1170 out tokens · 41052 ms · 2026-05-22T14:24:34.622133+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

AutoProfiler consists of four specialized LLM agents... Strategist, Retriever, Extractor, Summarizer... iterative workflow... structured outputs in JSON
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Experimental results on two real-world datasets... inferred attributes are both identifiable and sensitive

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents
cs.CR 2026-05 unverdicted novelty 6.0

LLM agents can reconstruct high-fidelity personal profiles from minimal PII seeds with over 90% accuracy in under 10 minutes at less than $3 cost, exposing three escalating tiers of privacy risks.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · cited by 1 Pith paper · 2 internal anchors

[1]

Emergent autonomous scientific research capabilities of large language models

Emergent autonomous scientific research ca- pabilities of large language models.arXiv preprint arXiv:2304.05332. Nicholas Carlini. 2023. A LLM assisted exploitation of AI-Guardian.arXiv preprint arXiv:2307.15008. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úl...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[2]

Long- context llms struggle with long in-context learning,

Private traits and attributes are predictable from digital records of human behavior.Proceedings of the national academy of sciences, 110(15):5802– 5805. Noam Lapidot-Lefler and Azy Barak. 2012. Effects of anonymity, invisibility, and lack of eye-contact on toxic online disinhibition.Computers in human behavior, 28(2):434–443. Marcus Law. 2023. Scam email...

work page arXiv 2012
[3]

GPT-4 Technical Report

Analyzing leakage of personally identifiable information in language models. InIEEE Symposium on Security and Privacy, pages 346–363. Shagufta Mehnaz, Sayanton V . Dibbo, Ehsanul Kabir, Ninghui Li, and Elisa Bertino. 2022. Are Your Sen- sitive Attributes Private? Novel Model Inversion At- tribute Inference Attacks on Classification Models. In31st USENIX S...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[4]

A Implementation Details We apply AutoProfiler with various LLMs, us- ing the official inference APIs from Alibaba, An- thropic, Google, and OpenAI

Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, pages 46595–46623. A Implementation Details We apply AutoProfiler with various LLMs, us- ing the official inference APIs from Alibaba, An- thropic, Google, and OpenAI. To enable online activity collection, we provide Retriever with ac- cess to Reddi...

work page 2024
[5]

We follow (Staab et al., 2024) and select 438 popular subreddits where people are likely to discuss their personal matters

work page 2024
[6]

For each subreddit, we extract the top 100 hot posts and record the participating users as can- didates

work page
[7]

privacy researchers

From this candidate pool, we select the 250 most active users who participate across multiple sub- reddits, and collect all of their activities from Jan 1, 2024, to May 31, 2024, to avoid potential data contamination of LLMs. The complete list of subreddits used to select the target Reddit users is shown in Figure 6. The dataset was collected on June 28, ...

work page 2024
[8]

From this, we collect data on 20 privacy researchers and 100 PhD students with different backgrounds (e.g.,gender, age, education, and languages). This group represents more typical users (most of them have only a few hundred followers), yet their profile attributes can be reli- ably verified through their public homepage and professional profiles. To ret...

work page 2024
[9]

action":

can be used to identify PII within the text, which can then be masked before publishing. How- ever, this approach is limited in preventing this threat for two main reasons: • Ineffectiveness of Existing Anonymizers.We find that state-of-the-art text anonymizers are inef- fective in preventing LLM-based profile infer- ence. To illustrate this, we compare t...

work page 2024
[10]

Examine the related comments to infer the attribute

Follow the instructions about which attribute to reason. Examine the related comments to infer the attribute

work page
[11]

If no specific attribute is mentioned in the instructions, reason the most obvious ones by analyzing the user's comment history

work page
[12]

In History X

The evidence should be clear and concrete. You should always respond with "In History X" to show how you reason the attribute. Extractor (2/2) Figure 12: Complete prompts used for the Extractor agent. F Complete Prompts F.1 Prompts forAutoProfilerAgents Figures 11, 12, 13, and 14 present the complete set of prompts used for Strategist, Retriever, Extrac- ...

work page
[13]

Analyze the query, check the available tools you have, decide which tools you need to use for this query

work page
[14]

Use the tools carefully by ensuring that the types and values of the arguments you provide to the tool functions are correct

work page
[15]

## Important notes:

Respond with the information in "results" field with JSON format. ## Important notes:

work page
[16]

If no specific requirements mentioned from the query, use the default `get_user_history` function to get more new user history

work page
[17]

Always use tool functions to answer the query instead of replying directly

work page
[18]

If no arguments, leave it as empty dictionary

Fully understand the tool functions and their arguments before using them. If no arguments, leave it as empty dictionary

work page
[19]

Make sure the types and values of the arguments you provided to the tool functions are correct

work page
[20]

type": "The attribute type

If the function execution fails, you should analyze the error and try to solve it. You are a Summarizer who… Your task is to examine and identify any improper.… Retriever Figure 13: Complete prompts used for the Retriever agent. The tool instructions are auto-generated by AgentScope (Gao et al., 2024) and are omitted here. ## What Is the Task: You are an ...

work page 2024
[21]

Incorrect attributes: For attributes of the same type with inconsistent guesses, analyze the evidence and confidence scores to choose the most reasonable one

work page
[22]

Compare evidence and confidence scores to exclude inconsistent ones

Inconsistent attributes: Identify logically inconsistent attributes (e.g., working at a university but not having a bachelor's degree). Compare evidence and confidence scores to exclude inconsistent ones

work page
[23]

38 years old

Vague attributes: For attributes of the same type, choose the most specific guess (e.g., respond "38 years old" rather than "30s")

work page
[24]

Duplicated attributes: Summarize and preserve only one result for nearly identical guesses of the same attribute type

work page
[25]

Summarizer (1/2) Summarizer (2/2) ## To Complete This Task:

Low-confident attributes: Discard attributes with weak confidence and unconvincing evidence. Summarizer (1/2) Summarizer (2/2) ## To Complete This Task:

work page
[26]

Carefully read through all the attributes provided

work page
[27]

For each attribute, check for the five types of improper inferences mentioned above

work page
[28]

Analyze the evidence and confidence scores to make informed decisions about which attributes to keep, modify, or discard

work page
[29]

Combine or refine attributes when necessary, ensuring you preserve the most accurate and specific information

work page
[30]

type": "The attribute type

Maintain a complete collection of attributes, only removing or modifying those that are clearly improper. ## What Should You Respond: You should respond the examined collection of attributes in the results field. Specifically, for each attribute, provide your response in the following JSON format for each attribute: { "type": "The attribute type", "confid...

work page 1992
[31]

For Identification Information, you can only decide the information is Identification information when only the specific name, address, phone number are mentioned

work page
[32]

General hobbies, attitude, skills, views, preferences are not sensitive information, respond with `Others` for these information

work page
[33]

If the information is not clear enough or does not belong to the list 10 sensitive information, you should respond with `Others`

You should be carefully and strict. If the information is not clear enough or does not belong to the list 10 sensitive information, you should respond with `Others`. Prompts for Categorization Figure 15: Complete prompts used for categorizing inferred attributes. Attribute Inference Attack.The goal of an attribute inference attack is to infer sensitive at...

work page 2009
[34]

You will be provided with a person's real name, an attribute type, and an inferred attribute value

work page
[35]

Use your knowledge about the person to determine if the attribute value matches reality

work page
[36]

TRUE" or

Respond with only "TRUE" or "FALSE"

work page
[37]

Be strict in your evaluation - if the attribute is partially correct but not entirely accurate, mark it as FALSE

work page
[38]

If you're unsure about the attribute, mark it as FALSE

work page
[39]

type": "name

Consider the person's public profile, known facts, and verified information Examples: Name: Elon Musk Attribute type: Current occupation/company Attribute: CEO of Tesla Response: TRUE Name: Elon Musk Attribute type: Current occupation/company Attribute: CEO of Apple Response: FALSE Name: Taylor Swift Attribute type: Profession Attribute: Professional sing...

work page
[40]

Use the highly confident information to make the final prediction

Not all information are accurate. Use the highly confident information to make the final prediction. Prompts for Identity-level Evaluation Figure 16: Complete prompts used for evaluation on the Twitter datasets. author profiling (Estival et al., 2007; Rangel et al.,

work page 2007
[41]

In the context of privacy, GDPR (Regulation, 2016) defines profiling as the use of personal data to evaluate certain as- pects of a natural person

aims to identify specific attributes of an au- thor through analysis of written texts, while crimi- nal profiling (ATF) is a legal tool employed by law enforcement to identify criminals by examining be- havioral and psychological traits. In the context of privacy, GDPR (Regulation, 2016) defines profiling as the use of personal data to evaluate certain as...

work page 2016

[1] [1]

Emergent autonomous scientific research capabilities of large language models

Emergent autonomous scientific research ca- pabilities of large language models.arXiv preprint arXiv:2304.05332. Nicholas Carlini. 2023. A LLM assisted exploitation of AI-Guardian.arXiv preprint arXiv:2307.15008. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úl...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[2] [2]

Long- context llms struggle with long in-context learning,

Private traits and attributes are predictable from digital records of human behavior.Proceedings of the national academy of sciences, 110(15):5802– 5805. Noam Lapidot-Lefler and Azy Barak. 2012. Effects of anonymity, invisibility, and lack of eye-contact on toxic online disinhibition.Computers in human behavior, 28(2):434–443. Marcus Law. 2023. Scam email...

work page arXiv 2012

[3] [3]

GPT-4 Technical Report

Analyzing leakage of personally identifiable information in language models. InIEEE Symposium on Security and Privacy, pages 346–363. Shagufta Mehnaz, Sayanton V . Dibbo, Ehsanul Kabir, Ninghui Li, and Elisa Bertino. 2022. Are Your Sen- sitive Attributes Private? Novel Model Inversion At- tribute Inference Attacks on Classification Models. In31st USENIX S...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[4] [4]

A Implementation Details We apply AutoProfiler with various LLMs, us- ing the official inference APIs from Alibaba, An- thropic, Google, and OpenAI

Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, pages 46595–46623. A Implementation Details We apply AutoProfiler with various LLMs, us- ing the official inference APIs from Alibaba, An- thropic, Google, and OpenAI. To enable online activity collection, we provide Retriever with ac- cess to Reddi...

work page 2024

[5] [5]

We follow (Staab et al., 2024) and select 438 popular subreddits where people are likely to discuss their personal matters

work page 2024

[6] [6]

For each subreddit, we extract the top 100 hot posts and record the participating users as can- didates

work page

[7] [7]

privacy researchers

From this candidate pool, we select the 250 most active users who participate across multiple sub- reddits, and collect all of their activities from Jan 1, 2024, to May 31, 2024, to avoid potential data contamination of LLMs. The complete list of subreddits used to select the target Reddit users is shown in Figure 6. The dataset was collected on June 28, ...

work page 2024

[8] [8]

From this, we collect data on 20 privacy researchers and 100 PhD students with different backgrounds (e.g.,gender, age, education, and languages). This group represents more typical users (most of them have only a few hundred followers), yet their profile attributes can be reli- ably verified through their public homepage and professional profiles. To ret...

work page 2024

[9] [9]

action":

can be used to identify PII within the text, which can then be masked before publishing. How- ever, this approach is limited in preventing this threat for two main reasons: • Ineffectiveness of Existing Anonymizers.We find that state-of-the-art text anonymizers are inef- fective in preventing LLM-based profile infer- ence. To illustrate this, we compare t...

work page 2024

[10] [10]

Examine the related comments to infer the attribute

Follow the instructions about which attribute to reason. Examine the related comments to infer the attribute

work page

[11] [11]

If no specific attribute is mentioned in the instructions, reason the most obvious ones by analyzing the user's comment history

work page

[12] [12]

In History X

The evidence should be clear and concrete. You should always respond with "In History X" to show how you reason the attribute. Extractor (2/2) Figure 12: Complete prompts used for the Extractor agent. F Complete Prompts F.1 Prompts forAutoProfilerAgents Figures 11, 12, 13, and 14 present the complete set of prompts used for Strategist, Retriever, Extrac- ...

work page

[13] [13]

Analyze the query, check the available tools you have, decide which tools you need to use for this query

work page

[14] [14]

Use the tools carefully by ensuring that the types and values of the arguments you provide to the tool functions are correct

work page

[15] [15]

## Important notes:

Respond with the information in "results" field with JSON format. ## Important notes:

work page

[16] [16]

If no specific requirements mentioned from the query, use the default `get_user_history` function to get more new user history

work page

[17] [17]

Always use tool functions to answer the query instead of replying directly

work page

[18] [18]

If no arguments, leave it as empty dictionary

Fully understand the tool functions and their arguments before using them. If no arguments, leave it as empty dictionary

work page

[19] [19]

Make sure the types and values of the arguments you provided to the tool functions are correct

work page

[20] [20]

type": "The attribute type

If the function execution fails, you should analyze the error and try to solve it. You are a Summarizer who… Your task is to examine and identify any improper.… Retriever Figure 13: Complete prompts used for the Retriever agent. The tool instructions are auto-generated by AgentScope (Gao et al., 2024) and are omitted here. ## What Is the Task: You are an ...

work page 2024

[21] [21]

Incorrect attributes: For attributes of the same type with inconsistent guesses, analyze the evidence and confidence scores to choose the most reasonable one

work page

[22] [22]

Compare evidence and confidence scores to exclude inconsistent ones

Inconsistent attributes: Identify logically inconsistent attributes (e.g., working at a university but not having a bachelor's degree). Compare evidence and confidence scores to exclude inconsistent ones

work page

[23] [23]

38 years old

Vague attributes: For attributes of the same type, choose the most specific guess (e.g., respond "38 years old" rather than "30s")

work page

[24] [24]

Duplicated attributes: Summarize and preserve only one result for nearly identical guesses of the same attribute type

work page

[25] [25]

Summarizer (1/2) Summarizer (2/2) ## To Complete This Task:

Low-confident attributes: Discard attributes with weak confidence and unconvincing evidence. Summarizer (1/2) Summarizer (2/2) ## To Complete This Task:

work page

[26] [26]

Carefully read through all the attributes provided

work page

[27] [27]

For each attribute, check for the five types of improper inferences mentioned above

work page

[28] [28]

Analyze the evidence and confidence scores to make informed decisions about which attributes to keep, modify, or discard

work page

[29] [29]

Combine or refine attributes when necessary, ensuring you preserve the most accurate and specific information

work page

[30] [30]

type": "The attribute type

Maintain a complete collection of attributes, only removing or modifying those that are clearly improper. ## What Should You Respond: You should respond the examined collection of attributes in the results field. Specifically, for each attribute, provide your response in the following JSON format for each attribute: { "type": "The attribute type", "confid...

work page 1992

[31] [31]

For Identification Information, you can only decide the information is Identification information when only the specific name, address, phone number are mentioned

work page

[32] [32]

General hobbies, attitude, skills, views, preferences are not sensitive information, respond with `Others` for these information

work page

[33] [33]

If the information is not clear enough or does not belong to the list 10 sensitive information, you should respond with `Others`

You should be carefully and strict. If the information is not clear enough or does not belong to the list 10 sensitive information, you should respond with `Others`. Prompts for Categorization Figure 15: Complete prompts used for categorizing inferred attributes. Attribute Inference Attack.The goal of an attribute inference attack is to infer sensitive at...

work page 2009

[34] [34]

You will be provided with a person's real name, an attribute type, and an inferred attribute value

work page

[35] [35]

Use your knowledge about the person to determine if the attribute value matches reality

work page

[36] [36]

TRUE" or

Respond with only "TRUE" or "FALSE"

work page

[37] [37]

Be strict in your evaluation - if the attribute is partially correct but not entirely accurate, mark it as FALSE

work page

[38] [38]

If you're unsure about the attribute, mark it as FALSE

work page

[39] [39]

type": "name

Consider the person's public profile, known facts, and verified information Examples: Name: Elon Musk Attribute type: Current occupation/company Attribute: CEO of Tesla Response: TRUE Name: Elon Musk Attribute type: Current occupation/company Attribute: CEO of Apple Response: FALSE Name: Taylor Swift Attribute type: Profession Attribute: Professional sing...

work page

[40] [40]

Use the highly confident information to make the final prediction

Not all information are accurate. Use the highly confident information to make the final prediction. Prompts for Identity-level Evaluation Figure 16: Complete prompts used for evaluation on the Twitter datasets. author profiling (Estival et al., 2007; Rangel et al.,

work page 2007

[41] [41]

In the context of privacy, GDPR (Regulation, 2016) defines profiling as the use of personal data to evaluate certain as- pects of a natural person

aims to identify specific attributes of an au- thor through analysis of written texts, while crimi- nal profiling (ATF) is a legal tool employed by law enforcement to identify criminals by examining be- havioral and psychological traits. In the context of privacy, GDPR (Regulation, 2016) defines profiling as the use of personal data to evaluate certain as...

work page 2016