Can a type provider be passed to a function as a parameter

Question

I am learning F # and FSharp.Data library . I have a task that I need to read 20 CSV files. Each file has a different number of columns, but the entries are of the same nature: based on a date string, and all other columns are floating point numbers. I need to do some statistical calculations in float format data columns before storing the results in a database. Although I have all the working logic:

• read in CSV through the FSHarp.Data CSV provider,
• use reflection to get the type of each field of the column along with the names of the headers that they give in accordance with a template that solves the corresponding calculation logic
• sqlbulkcopy result), I finished 20 functions (1 per CSV file).

The solution is far from acceptable. I thought I could create a generic top-level function so that the driver could iterate over all the files. However, after several days of trying, I get nothing.

A provider of type FSHarp.Data CSV has the following template:

type Stocks = CsvProvider<"../docs/MSFT.csv"> let msft = Stocks.Load("http://ichart.finance.yahoo.com/table.csv?s=MSFT") msft.Data |> Seq.map(fun row -> do something with row) ... 

I tried:

 let mainfunc (typefile:string) (datafile:string) = let msft = CsvProvider<typefile>.Load(datafile) .... 

This does not work, because CsvProvider complains that the file type is not a valid constant expression. I assume that the type provider needs the file to output the column type during encoding, the type inference cannot be delayed until the mainfunc code is called with the corresponding information.

Then I tried passing Type to mainfunc as a parameter

neither

 let mainfunc (typeProvider:CsvProvider<"../docs/MSFT.csv">) = .... 

and

 let mainfunc<typeProvider:CsvProvider<"../docs/MSFT.csv">> = .... 

worked.

Then I tried to transfer the MSFT from

 type Stocks = CsvProvider<"../docs/MSFT.csv"> let msft = Stocks.Load("http://ichart.finance.yahoo.com/table.csv?s=MSFT") 

In mainFunc. According to intellisence, MSFT is of type CsvProvider<...> and MSFT.Data is of type seq<CsvProvider<...>> . I tried to declare an input parameter with an explicit type of these two, but none of them can pass compilation.

Can anyone help and point me in the right direction? Did I miss something fundamental here? Any object of type .net and a class object can be used in the F # function to explicitly indicate the type of parameter, but can I do the same with the type from the type provider?

If the answer to the above question is not, what is the alternative to make a common logical file process 20 files or even 200 different files?

Answer 1

This is due to the type annotation for using the type F # TypeProvider, for example. FSharp.Data.JsonProvider <...> .DomainTypes.Url

Even if intellisense shows you CsvProvider<...> to refer to the msft type in the type annotation, you should use Stocks , and for msft.Data instead of CsvProvider<...>.Row you should use Stocks.Row .

If you want to do something dynamic, you can get the column names using msft.Headers , and you can get the column types using Microsoft.FSharp.Reflection.FSharpType.GetTupleElements(typeof<Stocks.Row>) (this works, because that the string is erased before the tuple at runtime)

EDIT:

If the formats are incompatible and you are dealing with dynamic data that does not match the general format, you can use CsvFile ( http://fsharp.imtqy.com/FSharp.Data/library/CsvFile.html ) instead, but you will lose all supplier type security. You can also use Deedle instead ( http://bluemountaincapital.imtqy.com/Deedle/ )