AlphaSharpe: LLM-Driven Discovery of Robust Risk-Adjusted Metrics

Hassan Sawaf; Kamer Ali Yuksel

REVIEW 4 major objections 6 minor 25 references

AlphaSharpe: LLM-Driven Discovery of Robust Risk-Adjusted Metrics

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-driven evolutionary loop can discover financial formulas that rank assets and build portfolios more accurately than the classic Sharpe ratio.

desk verdict Plausible LLM-driven metric discovery framework undermined by a selection-bias confound in the headline numbers; the 3x/2x claim is not yet supported. read the letter →

arxiv 2502.00029 v2 pith:BMPLRBX2 submitted 2025-01-23 q-fin.PM cs.AIcs.CLcs.NEq-fin.RM

Kamer Ali Yuksel , Hassan Sawaf This is my paper · ORCID

classification q-fin.PMcs.AIcs.CLcs.NEq-fin.RM

keywords LLM-drivendiscoveryrisk-adjustedmetricsSharperatioevolutionarysearchportfolioconstructionassetrankingcross-validationfinancialmachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an LLM-driven evolutionary search can discover new risk-return formulas that are materially better than the Sharpe ratio at predicting future performance and at building portfolios. The discovered AlphaSharpe metrics correlate with future Sharpe ratios at roughly three times the Spearman correlation of the Sharpe ratio itself, and portfolios built from their top-ranked assets show large Sharpe-ratio improvements on a 15-year US stock and ETF dataset. A reader should care because asset ranking and portfolio construction are core finance tasks, and the paper's metrics are closed-form formulas that can be applied directly.

What carries the argument

The carrying mechanism is the evolutionary metric-discovery loop: an LLM generates and mutates metric formulas using few-shot prompts, each candidate is scored by Spearman's rho, Kendall's tau, and NDCG against realized future Sharpe ratios within cross-validation folds, and only high-quality diverse candidates survive for the next round of crossover. The discovered alphaS1 through alphaS4 formula family is the concrete output, where alphaS1 emphasizes log-return compounding, alphaS2 adds downside risk and forecast volatility, alphaS3 adds skewness, kurtosis, and maximum drawdown, and alphaS4 adds a regime-dependent boost when average excess returns are positive.

What would settle it

Freeze the final alphaS1 through alphaS4 formulas, then evaluate them on a fresh market sample that was not part of the 15-year US dataset or its cross-validation folds; if their Spearman correlation with future Sharpe ratios is not substantially above the Sharpe ratio's own correlation on that fresh sample, the central 3x claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a closed loop—an LLM proposes candidate metric formulas through crossover and mutation, scores them by rank correlation with future Sharpe ratios on time-series cross-validation folds, and recombines the top scorers—produces formulas that outperform the Sharpe ratio and the probabilistic Sharpe ratio on both ranking and portfolio tests. The strongest empirical statements are Table 1, where alphaS4 reaches a Spearman correlation of 0.409 versus 0.130 for the Sharpe ratio, and Table 2, where alphaS2 portfolios improve test-period Sharpe by up to 93.97% and PSR by up to 101.99%. The discovered formulas replace the simple mean-over-standard-deviation structure with log excess returns, stability constants, downside-risk and forecast-volatility terms, higher-order moment adjustments, and a regime-dependent factor.

Load-bearing premise

The reported edge rests on the assumption that scoring candidate formulas by their correlation with future Sharpe ratios in cross-validation does not itself inflate those same out-of-sample correlations; without nested validation, selection bias could explain the gains.

Editorial extensions

If this is right

If the ranking correlations hold, portfolio managers can rank a candidate universe by an AlphaSharpe metric instead of the Sharpe ratio and expect a more accurate ordering of future risk-adjusted performance.
Portfolios selected from the top-ranked assets under alphaS2 are claimed to achieve test-period Sharpe improvements of roughly 76% to 102% relative to Sharpe-ratio-selected portfolios across top-10% to top-25% thresholds.
Because the evolved formulas are closed-form and open-sourced, they can be applied without retraining a model, making them immediately usable in existing ranking and allocation pipelines.
The paper claims the edge persists through stressed periods such as the 2020 COVID crash, since the formulas incorporate downside, tail, and regime adjustments.
The same LLM-driven evolution loop is also applied to portfolio weights, yielding an AlphaSharpe Portfolio that improves Sharpe by 71% and Calmar by 116% over equal weighting in the paper's comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Not tested in the paper: the search loop is the generalizable contribution, so running the same evolution with a different objective—such as correlation with drawdown-adjusted returns or transaction-cost-aware Sharpe—would likely produce different formulas and is a natural next experiment.
If the evolutionary selection is honest, the approach should transfer to other asset classes and time periods; if selection is overfit, the edge will shrink on repeated fresh samples, so a fully held-out market test is the decisive follow-up.
A practical consequence left implicit in the paper is that the formulas' advantage should be largest where Sharpe-ratio assumptions fail most, namely skewed and fat-tailed return distributions, so splitting the test sample by skewness would sharpen the comparison.
The regime indicator in alphaS4 is extremely simple—a step function on positive average excess returns—so it likely understates what a more flexible regime model could do, which the paper does not explore.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is worth a look: using an LLM to evolve financial risk-adjusted metrics through crossover, mutation, and scoring is a reasonable extension of program-search work like FunSearch, and it is new in the portfolio context as far as I know. The paper deserves credit for framing the problem clearly and for citing the right literatures, including Bailey and López de Prado on selection bias. The discovered formulas are interpretable and combine known building blocks (log returns, downside risk, higher moments, regime factors), which is sensible even if not deeply novel. The authors also say the code is open-sourced, which is good practice if it is actually available.

But the headline claims rest on an evaluation design that cannot support them. The metrics are evolved to maximize correlation with future Sharpe ratios inside cross-validation, and then the same Spearman statistic is reported on a single three-year test period. That is selecting on the outcome and then reporting the outcome. No nested evaluation, no selection-bias correction (despite citing the deflated Sharpe ratio), no confidence intervals, no multiple-testing adjustment. The reported 0.409 vs 0.130 Spearman correlation could easily be a winner's-curse artifact. The portfolio results are a single realized path with no bootstrap or repeated testing. These are load-bearing problems, not minor quibbles.

There are also smaller but real issues: the formula notation is garbled in places, several constants (epsilon, lambda, the regime threshold) are unreported, and the portfolio optimization section feels tacked on, with duplicated table numbers and no detail on how the method was discovered or validated. A rigorous reader would need the code and a precise description of the search and evaluation protocol.

All that said, the framework is plausible and worth engaging with. The central idea—using an LLM to propose and refine financial metrics with a scoring function tied to forward-looking performance—is not nonsense; it just needs honest evaluation. I would send this to a serious referee who knows both finance and evolutionary search, and I would require major revisions: nested or repeated evaluation, comparison to established robust metrics (Sortino, Omega, deflated Sharpe), full hyperparameters, and a clean statement of the selection procedure. If those are provided, the 3x/2x claim might or might not survive, but at least it would be testable. As it stands, the paper is a useful description of a method with an unverified headline result. It is not ready to act on, but it is ready to be refereed.

Referee Report

4 major / 6 minor

Summary. The paper proposes AlphaSharpe, an LLM-driven evolutionary framework that iteratively generates, mutates, and selects risk-adjusted performance metrics, using correlations with future Sharpe ratios as the fitness signal. The authors report four discovered metrics (αS1–αS4), with αS4 achieving Spearman 0.409 vs 0.130 for Sharpe/PSR on a 3-year out-of-sample period, and αS2-based portfolios improving Sharpe by up to 93.97% and PSR by 101.99%. A separate AlphaSharpe Portfolio allocation rule is also introduced and claimed to outperform risk parity and equal-risk-contribution portfolios. The dataset is 15 years of 3,246 US stocks and ETFs, with overlapping time-series cross-validation.

Significance. If the headline numbers were unbiased, the framework would be a useful contribution to q-fin.PM: it offers interpretable alternatives to Sharpe, claims an open-sourced implementation, and applies LLM-based evolutionary search in a novel domain. The paper also cites relevant prior work, including Bailey and López de Prado, and compares against PSR, which is appropriate. The main limitation is that the experimental design conflates model selection with model evaluation, and the formula/constant reporting is incomplete; these are fixable but currently block the central claims.

major comments (4)

[Sections 3.3 and 4, Table 1] The evaluation statistic is also the selection objective. Section 3.3 defines scoring by Spearman/Kendall/NDCG between historical metric scores and future Sharpe ratios, and Section 4 states that metrics were 'evolved based on their correlation with future Sharpe ratios within the cross-validation sets.' Table 1 then reports these same statistics on a single 20% test period. Under this design, the value 0.409 is not an unbiased estimate of predictive correlation; it is the best of a large candidate set. The paper cites Bailey and López de Prado (2014) on the deflated Sharpe ratio but does not apply any multiple-testing correction, nor does it provide a permutation null or nested evaluation. This is load-bearing for the '3x predictive power' claim. I request an outer evaluation loop that is never used for selection, or a deflated/selection-adjusted statistic, plus confidence intervals.
[Section 4.2, Table 2] The portfolio performance claim rests on a single realized test path. Table 2 reports improvements of up to +93.97% Sharpe and +101.99% PSR, but no standard errors, bootstrap intervals, or repeated rolling-window evaluations are provided. With four overlapping threshold variants and a metric family already selected on the same correlation objective, the '2x portfolio performance' claim has no measure of uncertainty. Please add at least a bootstrap over test-period returns and, ideally, an evaluation over multiple non-overlapping out-of-sample windows.
[Section 4, αS1–αS4 equations] The discovered metrics are not reproducibly specified. αS1's expression contains an unexplained 'q' and an ambiguous fraction; αS2's DR formula is missing a visible denominator for the second term; αS3 uses K, S, and MDD without definitions or numeric values; αS4 defines a regime boost with an arbitrary 0.1 constant; and the forecasted-volatility window in αS2 is written with 'n/4' but n is not defined. Since these constants are part of the fitted metric, their values must be reported. As written, the formulas cannot be reimplemented and the evolved metrics are not a well-defined contribution.
[Section 7, Tables 3 and 4] The AlphaSharpe Portfolio claim is not yet evaluable. The allocation rule is presented as 'newly discovered' but no selection procedure is described, so it may have been chosen using the same test data; key hyperparameters λ and ε are unreported. In addition, Tables 3 and 4 have identical captions and content, while the text refers only to Table 4. This second empirical claim should either be removed or given the same nested-evaluation treatment as Section 4.

minor comments (6)

[Section 4, dataset description] The dataset description should state the universe construction, return frequency, handling of delisted stocks, and the data provider; '3,246 US stocks and ETFs' alone is insufficient for reproducibility.
[Section 4, cross-validation details] The time-series cross-validation details are incomplete: number of folds, window lengths, and the amount of overlap are not quantified.
[Sections 4 and 7, open-source claim] The paper states that code is open-sourced, but no repository URL, DOI, or artifact identifier is provided.
[Section 4, αS2 formula] The summation in the forecasted-volatility term V is ambiguous; please clarify the lower limit and the definition of n.
[Section 4, αS4 formula] The indicator function in αS4 is typeset with an unusual symbol; please use standard notation such as 1{E[R−rf] > 0}.
[Tables 3 and 4] Tables 3 and 4 are duplicates; renumber them and make the text references consistent.

Circularity Check

1 steps flagged · score 6.0 of 10

Selection objective equals evaluation statistic: Table 1's 0.409 Spearman is a selected maximum, not an independent prediction.

fitted input called prediction [Section 4 (Experiments) and Table 1; Section 3.3 (Scoring Functions)]
"Metrics evolved based on their correlation with future Sharpe ratios within the cross-validation sets, ensuring robust evaluation across different periods."

The metric formulas are selected by evolutionary search using the scoring functions defined in Section 3.3, which measure Spearman/Kendall/NDCG correlation between historical metric scores and future Sharpe ratios. Table 1 then reports the same Spearman correlation on the out-of-sample test as evidence of '3x predictive power.' Because the LLM pipeline generates and mutates many candidates and retains those with the highest correlation on the cross-validation folds, the reported test value is the maximum of a searched distribution, not an unbiased estimate of predictive skill.

full rationale

The paper's central empirical claim—that evolved metrics have 3x predictive power and 2x portfolio performance—is not a self-contained derivation; it is the output of an optimization loop whose objective is the same statistic used for evaluation. Section 3.3 defines metric quality by correlation with future Sharpe ratios, and Section 4 states that metrics were evolved to maximize that correlation within cross-validation. Table 1 then reports Spearman correlation on a single out-of-sample test as if it were an independent measure of predictive power. Since many candidate formulas are tried and discarded, the reported 0.409 is a selected maximum and will be inflated even if the final test period was never touched. The paper cites Bailey and Lopez de Prado (2014) on the deflated Sharpe ratio but does not apply any multiple-testing correction or report confidence intervals, reinforcing that the selection bias is unaddressed. The portfolio results in Table 2 are a downstream consequence of ranking assets by a metric explicitly optimized to rank assets by future Sharpe, so they do not constitute an orthogonal validation. No self-citation load-bearing circularity is present; the issue is that the prediction reduces to the fitted selection objective, warranting a partial-circularity score of 6 rather than a full 10.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central empirical claims rest on the evaluation protocol and on unstated constants in the evolved formulas. The paper pulls from prior literature the idea that rank correlation with future Sharpe is a valid target, and from the LLM search the specific formulas, but it does not provide independent evidence for the chosen constants or for the absence of selection bias.

free parameters (6)

epsilon stability constants in alphaS1/alphaS2 = not reported
Added inside denominators and the downside-risk term to reduce sensitivity; value chosen by the search or by hand, not derived.
kurtosis, skewness, and drawdown coefficients in alphaS3 = 1/12, 1/6, 1
Fixed weights in the higher-moment adjustment; no fitting procedure or sensitivity analysis reported.
regime boost constant and threshold in alphaS4 = 0.1 and sign of mean excess return
Arbitrary multiplier and indicator threshold; appears selected by the evolutionary search.
forecasted volatility window in alphaS2 = n/4 (last quarter of sample)
Choice of window length for the forward-looking volatility term; no justification given.
covariance regularization lambda in AlphaSharpe Portfolio = not reported
Added to the covariance matrix before inversion; value not specified.
entropy regularization term in AlphaSharpe Portfolio = e^(-H), H = -sum_i w_i log(w_i + eps)
The weight update uses current weights in defining H, creating an implicit hyperparameter; exact implementation is unclear.

assumptions (7)

domain assumption The LLM search space of metric formulas is rich enough to contain a genuinely robust metric.
Section 3.2 assumes LLMs can propose innovative, valid financial metrics using implicit domain knowledge.
domain assumption Spearman, Kendall, and NDCG correlation with future Sharpe ratio is an appropriate objective for predictive power.
Section 3.3 uses these as scoring functions; this equates predictive power with rank alignment to future Sharpe.
domain assumption The 3-year out-of-sample period is representative of future regimes and includes expected stress.
Section 4 uses a single held-out test period to support generalization claims.
domain assumption Overlapping time-series cross-validation folds give unbiased rankings without leakage.
Section 4 describes overlapping folds but does not analyze leakage or autocorrelation effects.
domain assumption No transaction costs, borrow costs, or capacity constraints in portfolio backtests.
Section 4.2 constructs uniform-weight portfolios and evaluates Sharpe without costs; real-world performance may differ.
domain assumption Log returns are defined consistently (log(1+r) or log price ratios) for all assets.
The paper does not specify the exact definition used in alphaS1 through alphaS4, yet the formulas depend on it.
ad hoc to paper The constants in the evolved formulas are fixed across assets and time.
The final formulas use fixed constants (epsilon, 12, 6, 0.1) with no fitting or meta-analysis, implying stationarity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AlphaSharpe: LLM-Driven Discovery of Robust Risk-Adjusted Metrics." pith.science (2026). https://pith.science/paper/BMPLRBX2

@misc{pith2026250200029,
  author       = {Pith},
  title        = {Pith review of: AlphaSharpe: LLM-Driven Discovery of Robust Risk-Adjusted Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMPLRBX2}},
  note         = {Machine review of arXiv:2502.00029}
}

read the original abstract

Financial metrics like the Sharpe ratio are pivotal in evaluating investment performance by balancing risk and return. However, traditional metrics often struggle with robustness and generalization, particularly in dynamic and volatile market conditions. This paper introduces AlphaSharpe, a novel framework leveraging large language models (LLMs) to iteratively evolve and optimize financial metrics to discover enhanced risk-return metrics that outperform traditional approaches in robustness and correlation with future performance metrics by employing iterative crossover, mutation, and evaluation. Key contributions of this work include: (1) a novel use of LLMs to generate and refine financial metrics with implicit domain-specific knowledge, (2) a scoring mechanism to ensure that evolved metrics generalize effectively to unseen data, and (3) an empirical demonstration of 3x predictive power for future risk-returns, and 2x portfolio performance. Experimental results in a real-world dataset highlight the superiority of discovered metrics, making them highly relevant to portfolio managers and financial decision-makers. This framework not only addresses the limitations of existing metrics but also showcases the potential of LLMs in advancing financial analytics, paving the way for informed and robust investment strategies.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

[1]

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. 1999. Coherent measures of risk. Mathematical Finance, 9(3):203--228

work page 1999
[2]

David H Bailey and Marcos L \'o pez de Prado. 2012. The sharpe ratio efficient frontier. Journal of Risk, 15(2):3

work page 2012
[3]

David H Bailey and Marcos L \'o pez de Prado. 2014. The deflated sharpe ratio: Correcting for selection bias, backtest overfitting, and non-normality. The Journal of Portfolio Management, 40(5):94--107

work page 2014
[4]

Antoine Cully and Yiannis Demiris. 2017. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245--259

work page 2017
[5]

Victor DeMiguel, Lorenzo Garlappi, and Raman Uppal. 2009. Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy? The review of Financial studies, 22(5):1915--1953

work page 2009
[6]

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bogdan Kosti \'c , Radek Krej \'c \' r \' k, Peter Lajko, Nenad Toma s ev, Isabel von Glehn, et al. 2022. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47--53

work page 2022
[7]

James D Hamilton. 1989. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357--384

work page 1989
[8]

a rvelin and Jaana Kek \

Kalervo J \"a rvelin and Jaana Kek \"a l \"a inen. 2002. Cumulated gain-based evaluation of ir techniques. In ACM Transactions on Information Systems (TOIS), volume 20, pages 422--446. ACM

work page 2002

Show all 25 references

[9]

Con Keating and William F Shadwick. 2002. A universal performance measure. Journal of Performance Measurement, 6(3):59--84

2002
[10]

John L Kelly. 1956. A new interpretation of information rate. The Bell System Technical Journal, 35(4):917--926

1956
[11]

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81--93

1938
[12]

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Alexander L Gaunt. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

2022
[13]

Alex Lipton and Marcos Lopez de Prado. 2020. Three quant lessons from covid-19. Risk Magazine, April

2020
[14]

Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292

2024 arXiv
[15]

Sébastien Maillard, Thierry Roncalli, and Jérôme Teiletche. 2010. The properties of equally weighted risk contribution portfolios. The Journal of Portfolio Management, 36(4):60--70

2010
[16]

Harry Markowitz. 1952. http://www.jstor.org/stable/2975974 Portfolio selection . The Journal of Finance, 7(1):77--91

1952
[17]

Edward Qian. 2005. Risk parity portfolios: Efficient portfolios through true diversification. Panagora Asset Management, 1(1):1--10

2005
[18]

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models. Nature, 62...

2024
[19]

William F Sharpe. 1966. Mutual fund performance. Journal of business, pages 119--138

1966
[20]

Frank A Sortino and Lee N Price. 1994. Performance measurement in a downside risk framework. The Journal of Investing, 3(3):59--64

1994
[21]

Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72--101

1904
[22]

Terry Young. 1991. Calmar ratio: A smoother tool. Futures, 20(8):40--41

1991
[23]

Valeriy Zakamulin and Patricia Giner. 2015. Beyond the sharpe ratio: An application of higher moments for performance evaluation. Journal of Asset Management, 16(4):222--246

2015
[24]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[25]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. 1999. Coherent measures of risk. Mathematical Finance, 9(3):203--228

work page 1999

[2] [2]

David H Bailey and Marcos L \'o pez de Prado. 2012. The sharpe ratio efficient frontier. Journal of Risk, 15(2):3

work page 2012

[3] [3]

David H Bailey and Marcos L \'o pez de Prado. 2014. The deflated sharpe ratio: Correcting for selection bias, backtest overfitting, and non-normality. The Journal of Portfolio Management, 40(5):94--107

work page 2014

[4] [4]

Antoine Cully and Yiannis Demiris. 2017. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245--259

work page 2017

[5] [5]

Victor DeMiguel, Lorenzo Garlappi, and Raman Uppal. 2009. Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy? The review of Financial studies, 22(5):1915--1953

work page 2009

[6] [6]

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bogdan Kosti \'c , Radek Krej \'c \' r \' k, Peter Lajko, Nenad Toma s ev, Isabel von Glehn, et al. 2022. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47--53

work page 2022

[7] [7]

James D Hamilton. 1989. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2):357--384

work page 1989

[8] [8]

a rvelin and Jaana Kek \

Kalervo J \"a rvelin and Jaana Kek \"a l \"a inen. 2002. Cumulated gain-based evaluation of ir techniques. In ACM Transactions on Information Systems (TOIS), volume 20, pages 422--446. ACM

work page 2002