Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Computing Estimators of Dantzig Selector type via Column and Constraint Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read This paper shows that the Dantzig selector, basis pursuit, and fused Dantzig selector—three sparse-signal recovery problems normally too large for general LP solvers—can be solved at scales of $n\approx 10^3$ and $p\approx 10^6$ in…

desk verdict Useful large-scale Dantzig selector / basis pursuit solver, but the fused Dantzig section has a wrong inverse matrix that invalidates those results as written. read the letter →

arxiv 1908.06515 v1 pith:BA2QSUYJ submitted 2019-08-18 stat.CO math.OCstat.ML

classification stat.COmath.OCstat.ML MSC 90C0590C0662J07
keywords DantzigselectorbasispursuitfusedcolumngenerationconstraintsimplexwarmstartsLassoinitializationsparsesignalrecovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that sparse-signal estimation problems usually treated as hard large-scale linear programs become routinely solvable when two classical LP techniques are revived. Instead of handing the full problem to a solver, the method solves small subproblems over few selected columns and few selected constraints, expanding them only when optimality conditions fail. Initializing those subproblems with the solution of the closely related Lasso keeps the number of expansions small. If the argument holds, Dantzig-selector-type estimators that were impractical at million-variable scale become usable in minutes.

What carries the argument

The machinery is a reduced master LP coupled with pricing and separation. For the $\ell_1$-DS problem, the full LP is replaced by $\mathrm{DS}(I,J)$, which optimizes only $\beta_j$ for $j\in J$ while enforcing only the constraints indexed by $I$. After each solve, dual information gives reduced costs $|X_{*,j}^T\alpha|>1$, which identify columns to add, and primal residuals $|X_{*,i}^T r|>\lambda$, which identify violated constraints to add. The Lasso solution supplies the initial $I$ and $J$, and simplex warm starts carry basis information between subproblems. The same separation-and-pricing loop solves basis pursuit by column generation alone, and the fused Dantzig selector after a transformation that keeps the constraint matrix banded and makes each reduced-cost or constraint check cost $O(1)$.

What would settle it

Construct an $\ell_1$-DS instance with $n\approx 10^3$, $p\approx 10^6$ in which the true solution has about $p/2$ nonzero entries of comparable size, run DantzigLP with the Lasso warm start, and record total simplex iterations and runtime: if it still finishes in minutes the claim is stronger than the paper argues, while if it takes as long as or longer than solving the full LP, the paper's efficiency claim is limited to sparse solutions.

Watch

Extended reading notes

Core claim

The central claim is that column generation and constraint generation, run jointly inside a simplex LP solver and warmed up by a Lasso solution, produce accurate solutions to $\ell_1$-Dantzig selector problems at scales that commercial interior-point or first-order methods cannot reach: about $p\approx 10^6$ variables with $n\approx 10^3$ for the Dantzig selector, $p\approx 10^5$ for basis pursuit, and $n=p\approx 10^6$ for the fused Dantzig selector, all in minutes. The method relies on the optimal signal being sparse and on the active constraint set being small. The paper formulates a reduced LP over selected variable indices and selected constraint indices, checks dual reduced costs to add columns and primal violations to add constraints, and iterates until exact optimality conditions hold. It reports that the Lasso initialization is essential: random initialization, or column generation alone without constraint generation, loses most of the speed advantage.

Load-bearing premise

The whole speedup rests on the assumption that the Dantzig selector solution is sparse and that the Lasso solution already identifies most of its nonzero entries and active constraints; when that fails, column and constraint generation must add many pieces and the method slows toward solving the full LP.

Editorial extensions

If this is right

  • The $\ell_1$-Dantzig selector becomes solvable in minutes at $n\approx 10^3$, $p\approx 10^6$, with memory requirements far below methods that form $X^TX$.
  • Basis pursuit with $p\approx 10^5$ is solvable in minutes, while full-LP and ADMM baselines either exceed memory limits or fail to converge at that scale.
  • The fused Dantzig selector in the signal-estimation case $X=I$ scales to $n=p\approx 10^6$, and the regression case with general $X$ improves on solving the full LP.
  • Computing a whole path of solutions over a grid of $\lambda$ values is efficient because each solved subproblem warm-starts the next.
  • The Lasso initialization, not column generation by itself, is the key to the speed: removing it or replacing it with random initialization degrades performance substantially.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pricing-and-separation loop should extend to other LP-based sparse estimators, such as group or ordered Dantzig variants, whenever a cheap surrogate provides a reasonable initial support; this is a direct extrapolation of the paper's mechanism, not a claim it tests.
  • The paper's own runtime table shows DantzigLP degrading sharply as $\lambda$ shrinks and the solution densifies; read literally, the practical scope of the method is bounded by solution sparsity, not simply by $n$ and $p$.
  • Because the framework returns both primal and dual information at every iteration, a user could stop early with a certificate of suboptimality, trading exactness for large time savings on massive instances; the paper mentions the certificate but does not develop a stopping rule that uses it.
  • The reported wall-clock times depend on a warm-start-friendly simplex engine, so the advantage is not inherent to column generation alone and may not transfer to interior-point or first-order subsolvers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper develops DantzigLP, a column- and constraint-generation framework for solving large-scale linear-programming formulations of three sparse-estimation problems: the ℓ1-Dantzig selector, basis pursuit, and the fused Dantzig selector. The method solves reduced LPs over a subset of columns and constraints, checks optimality through dual violations and reduced costs, and warm-starts from solutions of related Lasso-type problems. The authors report experiments on synthetic and two real datasets showing large speedups over ADMM implementations, the parametric simplex method (PSM), and direct LP solves with Gurobi, and they provide a Julia implementation.

Significance. If the results hold, the paper addresses a real gap: LP-based Dantzig-type estimators are statistically attractive but are rarely used at scale because general-purpose LP solvers struggle with p in the millions. The idea of using classical column and constraint generation with a simplex solver and a Lasso initialization is clean and well motivated, and the pseudocode is precise enough to reimplement. The ℓ1-Dantzig selector and basis pursuit experiments support the central efficiency claim for sparse regimes, and the code release is a concrete strength. However, the fused Dantzig selector section contains a mathematical error in the claimed inverse matrix H, so the fused results in Tables 6 and 7 are not supported as written; in addition, the introduction's "few minutes" scalability claim is inconsistent with the 3,702-second Boston1M result in Table 4.

major comments (3)
  1. [§2.4, Eq. (14)] The matrix H defined by H_{i,j}=1 if j=1, H_{i,j}=i-j if i>j, and 0 otherwise is not the inverse of D=[e_1^T; D^(0)]. For p=4, the fourth column of H is zero and (DH)_{2,4}=0, so DH is not the identity; the correct inverse is the lower-triangular matrix of ones. Because this H is used in Eqs. (15)-(17), the derivation of the fused Dantzig selector LP is not valid: the constraints g_A=0 and g_B in [-lambda, lambda] in Eq. (17) do not represent H_A^T r=0 and ||H_B^T r||_inf <= lambda from Eq. (16). The regression reduction in §2.4.2 inherits the same issue, and Tables 6 and 7 therefore do not demonstrate solutions of (7) or (16) as written. Please correct H, rederive the reformulation, and either confirm that the reported experiments used the corrected formulation or regenerate the fused Dantzig experiments.
  2. [Introduction and Table 4] The introduction claims that DantzigLP can solve instances of the ℓ1-DS problem with n≈10^3 and p≈10^6 "within a few minutes," but the only p≈10^6 experiment reported, Boston1M in Table 4 with n=200, takes 3,702 seconds (about 62 minutes), and no synthetic instance with p≈10^6 is shown. As written, this scalability claim is not supported by the evidence. Please either temper the claim, distinguish the regimes in which the stated runtimes are observed, or provide an experiment with n≈10^3 and p≈10^6 that completes in the claimed time.
  3. [§2.2 and Table 3] The efficiency of DantzigLP depends on the premise that the Dantzig selector solution is sparse and close to the Lasso support. Table 3 shows this dependence clearly: DantzigLP (Path) rises from 13.6 seconds at tau=1.0 to 209 seconds at tau=0.1 as the solution becomes denser, and the text acknowledges that DantzigLP can be slower than the full LP for small p. Because this is a central boundary condition on the method, the abstract and conclusion should state the sparsity regime explicitly rather than relying on the phrase "in many settings."
minor comments (6)
  1. [Tables 1–3, 5, 7] Runtimes are reported only as means over replications; given the variability typical of simplex-based and column/constraint-generation methods, reporting standard deviations or median/interquartile ranges would make the comparisons substantially more informative.
  2. [§2.1] The sentence "Combining column and constraint generation LP techniques with methods for finding good initializations for the initial column and constraint sets I and J, we develop DantzigLP, a general framework for solving large-scale versions of the the Dantzig Selector-type problems" contains a duplicated "the."
  3. [§2.4] The heading "F used Dantzig Selector" contains an extra space and should be "Fused Dantzig Selector."
  4. [§3.3] The sentence "the initial solution obtained from the Lasso problem (5)" should refer to the fused Lasso problem (5) rather than the Lasso problem (3), to avoid confusion between the two regularizers.
  5. [Algorithms 2–3 and §2.2] The pseudocode in Algorithms 2 and 3 uses exact violation checks, while Section 2.2 defines a relaxed version with tolerance epsilon=10^{-4} that is used in the experiments; the pseudocode should incorporate this tolerance to match the implementation.
  6. [Table 4] The DantzigLP runtimes include the Lasso initialization, but the table reports only the combined time; decomposing the runtime into initialization versus column/constraint generation would clarify where the 3,702 seconds for Boston1M is spent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Lasso warm start is an initialization, not a fitted prediction, and all LP solutions are checked against full optimality conditions.

full rationale

The paper's central derivation—column and constraint generation applied to LP formulations of l1-DS, Basis Pursuit, and the Fused Dantzig Selector—does not reduce to its inputs. The Lasso solution is used only to initialize the index sets I and J (Algorithm 2) and the Basis Pursuit starting support (Algorithm 3); every reported solution is accepted only after the full optimality checks Iv=∅ and Jv=∅ (Section 2.1 and eq. (12)). Thus the Lasso is a warm start, not a fitted parameter disguised as a prediction, and the runtime claims are measured outcomes (Tables 1–7), not consequences of a definitional identity. The only self-citation, [12] (the authors' prior SVM column/constraint-generation paper), is contextual and non-load-bearing: the DS LP formulations, optimality conditions, and numerical comparisons with Gurobi, PSM, and ADMM stand independently. The cited Lasso–DS path equivalence results [20] and [1] are external works. Section 2.2's discussion that efficiency depends on sparsity and on the Lasso's similarity to DS is a stated assumption about computational performance, not a circular derivation; Table 3 even shows the method slowing when the assumption weakens. The suspicious H=D^{-1} formula in Section 2.4 (which would make DH≠I) is a mathematical correctness issue that may undermine Tables 6–7, but it is not a circularity, so it is not scored here. Overall: no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard LP machinery plus two empirical premises about sparsity and Lasso-DS similarity. The only hand-tuned quantities are algorithm hyperparameters (tolerances and batch sizes); no new statistical entities are postulated.

free parameters (3)
  • Violation tolerance epsilon = 1e-4
    Used in Eq. (12) to relax column and constraint generation violation checks; chosen by the authors, affects number of iterations and solution accuracy.
  • Max columns added per iteration = 30
    Section 3.1: 'we add up to 30 columns with the most negative reduced costs'; a hand-tuned batch size affecting runtime.
  • Max constraints added per iteration = 50
    Section 3.1: 'and up to 50 of the most violated constraints'; a hand-tuned batch size affecting runtime.
assumptions (3)
  • standard math LP duality and simplex optimality conditions
    Used throughout Section 2.1 to justify extending reduced primal and dual solutions to the full LP; standard background.
  • domain assumption Lasso solution provides a good approximation to the Dantzig selector support and active constraint set
    Section 2.2, around Eq. (11); this is an empirical premise on which the initialization of Algorithms 2 and 3 depends. The paper cites [20, 1] for conditions of path equivalence, but these do not hold generally.
  • domain assumption The optimal Dantzig selector solution is sparse
    The column generation strategy only works if few nonzero components exist; the paper states 'at optimality, an optimal beta will have few nonzeros' (Section 1.1). This is standard for such estimators but not universally guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Computing Estimators of Dantzig Selector type via Column and Constraint Generation." pith.science (2026). https://pith.science/paper/BA2QSUYJ

@misc{pith2026190806515,
  author       = {Pith},
  title        = {Pith review of: Computing Estimators of Dantzig Selector type via Column and Constraint Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BA2QSUYJ}},
  note         = {Machine review of arXiv:1908.06515}
}
read the original abstract

We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements. Specific formulations of the reconstruction problem considered here include Dantzig selector, basis pursuit (for the case in which the measurements contain no errors), and the fused Dantzig selector (for the case in which the underlying signal is piecewise constant). In spite of being estimators central to sparse signal processing and machine learning, solving these linear programming problems for large scale instances remains a challenging task, thereby limiting their usage in practice. We show that classic constraint- and column-generation techniques from large scale linear programming, when used in conjunction with a commercial implementation of the simplex method, and initialized with the solution from a closely-related Lasso formulation, yields solutions with high efficiency in many settings.

Figures

Figures reproduced from arXiv: 1908.06515 by the authors.

Figure 1
Figure 1. Feasibility and objective violations as a function of runtime for DantzigLP (including time required for Lasso initialization) and the ADMM implementation Flare, for the `1-DS problem. In all instances, DantzigLP reaches an optimal solution with zero feasibility violation within a few iterations. In comparison, Flare takes much longer to improve the objective and feasibility violations, often failing to converge eve… view at source ↗
Figure 2
Figure 2. Comparison of 5 variants of DantzigLP (n = 1, 000, p = 10, 000), showing the importance of all components of the approach: Lasso initialization, column generation, constraint generation, and the simplex solver. 3.1.2 Experiments on real datasets We demonstrate the performance of DantzigLP on real-datasets with p ≈ 106 and n ≈ 103 . Due to memory constraints (our maximum was 64GB), only DantzigLP could solve these pr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 30 canonical work pages

  1. [1]

    M. S. Asif and J. Romberg. On the lasso and dantzig selector equivalence. In 2010 44th Annual Conference on Information Sciences and Systems (CISS) , pages 1–6, 3 2010

  2. [2]

    M. S. Asif and J. Romberg. Dantzig selector homotopy with dynamic measurements. Proc. SPIE 7246, Computational Imaging VII, 72460E , 7246, 2009

  3. [3]

    Beck and M

    A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences , 2(1):183–202, 2009

  4. [4]

    S. R. Becker, E. J. Cand` es, and M. C. Grant. Templates for convex cone problems with applications to sparse signal recovery. Mathematical Programming Computation, 3(3):165, 7 2011

  5. [5]

    Bertsimas and J

    D. Bertsimas and J. N. Tsitsiklis. Introduction to Linear Optimization . Athena, 1997

  6. [6]

    P. J. Bickel, Y. Ritov, and A. B. Tsybakov. Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics , 37(4):1705–1732, 2009

  7. [7]

    S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statisti- cal learning via the alternating direction method of multipliers. Found. Trends Mach. Learn., 3(1):1–122, January 2011

  8. [8]

    Brauer, D

    C. Brauer, D. A. Lorenz, and A. M. Tillmann. A primal-dual homotopy algorithm for 𝓁1- minimization with 𝓁∞-constraints. Computational Optimization and Applications , 70(2):443– 478, 2018

Show all 31 references
  1. [9]

    Candes and T

    E. Candes and T. Tao. The dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist., 35(6):2313–2351, 12 2006

  2. [10]

    Chen and D

    S. Chen and D. Donoho. Basis pursuit. In Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers , volume 1, pages 41–44 vol.1, 10 1994

  3. [11]

    G. B. Dantzig and P. Wolfe. Decomposition principle for linear programs. Operations research, 8(1):101–111, 1960

  4. [12]

    Dedieu and R

    A. Dedieu and R. Mazumder. Solving large-scale l1-regularized svms and cousins: the sur- prising effectiveness of column and constraint generation. arXiv preprint arXiv:1901.01585 , 2019

  5. [13]

    D. L. Donoho, A. Maleki, and A. Montanari. Message-passing algorithms for compressed sensing. Proceedings of the National Academy of Sciences , 106(45):18914–18919, 2009

  6. [14]

    Efron, T

    B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. The Annals of statistics, 32(2):407–499, 2004

  7. [15]

    Efron, T

    B. Efron, T. Hastie, and R. Tibshirani. Discussion: The dantzig selector: Statistical estimation when p is much larger than n. The Annals of Statistics , 35(6):2358–2364, 2007. 21

  8. [16]

    L. R. Ford Jr and D. R. Fulkerson. A suggested computation for maximal multi-commodity network flows. Management Science, 5(1):97–101, 1958

  9. [17]

    Friedman, T

    J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software , 33(1):1–22, 2010

  10. [18]

    Harrison and D

    D. Harrison and D. L. Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management , 5(1):81 – 102, 1978

  11. [19]

    Hazimeh and R

    H. Hazimeh and R. Mazumder. Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms. ArXiv e-prints, March 2018

  12. [20]

    G. M. James, P. Radchenko, and J. Lv. Dasso: connections between the dantzig selector and lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 71(1): 127–142, 2009

  13. [21]

    N. A. Johnson. A dynamic programming algorithm for the fused lasso and l 0-segmentation. Journal of Computational and Graphical Statistics , 22(2):246–260, 2013

  14. [22]

    X. Li, T. Zhao, X. Yuan, and H. Liu. The flare package for high dimensional linear regression and precision matrix estimation in r. Journal of Machine Learning Research, 16:553–557, 2015

  15. [23]

    Z. Lu, T. K. Pong, and Y. Zhang. An alternating direction method for finding dantzig selectors. Computational Statistics & Data Analysis , 56(12):4037 – 4046, 2012

  16. [24]

    Mammen and S

    E. Mammen and S. Geer. Locally adaptive regression splines. The Annals of Statistics , 25(1): 387–413, 1997

  17. [25]

    H. Pang, H. Liu, and R. Vanderbei. The fastclime package for linear programming and large- scale precision matrix estimation in r. Journal of Machine Learning Research , 15(1):489–493, January 2014

  18. [26]

    H. Pang, H. Liu, R. J. Vanderbei, and T. Zhao. Parametric simplex method for sparse learning. In Advances in Neural Information Processing Systems , pages 188–197, 2017

  19. [27]

    L. I. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena , 60(1):259 – 268, 1992

  20. [28]

    Tibshirani

    R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267–288, 1996

  21. [29]

    Tibshirani, M

    R. Tibshirani, M. Saunders, S. Rosset, J. Zhu, and K. Knight. Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society Series B , pages 91–108, 2005

  22. [30]

    Wang and X

    X. Wang and X. Yuan. The linearized alternating direction method of multipliers for dantzig selector. SIAM Journal on Scientific Computing , 34(5):A2792–A2811, 2012

  23. [31]

    S. J. Wright, R. D. Nowak, and M. A. Figueiredo. Sparse reconstruction by separable approx- imation. IEEE Transactions on Signal Processing , 57(7):2479–2493, 2009. 22

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.