Discriminant fishing time

Question

I recently implemented Fisher Linear Discriminant (FLD) in Lisp. So far I have used dim (<10) samples with populations of the order of 10 with which FLD was performed immediately. This morning I used real-world data with dim (5) and populations of the order of 10 ⁴ and the program has been running for several hours on my AMD Athlon (tm) II Dual-Core M320 × 2 laptop. Here are the file sizes that I use:

$ wc output_sig.txt 13000 65000 627677 output_sig.txt $ wc output_bkg.txt 13000 65000 644621 output_bkg.txt 

In light of the above, how is the calculation time of the FLD scale calculated with the sample size and dimension? Can we expect several hours of calculation here?

Answer 1

Without knowledge of the code: LDA is basically a generalized eigenvalue problem, so the problem is resolved if you use a fairly efficient routine of linear algebra. Typically, handwritten routines in LISP are not very efficient, so I recommend using a LAPACK wrapper such as Matlisp.