Are there any ETL tools that integrate with Rails models?

Question

I am learning ETL tools for importing flat files into a database and then exporting xml files.

Many of the tools support generating code for use in your application; however, I did not find any support using the code already in your application. Our model is complex (relationships, checks, polymorphic associations, callbacks, etc.).

What tools are available to reuse existing code? Or am I stuck recreating (and supporting) my model in an ETL tool?

Note. My ETL requirements (as opposed to bulk inserts or activerecord-import) are conversions. We receive data from more than 200 different sources in various formats, levels of completeness and purity. In addition, the most “designer” is more realistic for less technical users who will define the transformations.

Answer 1

ActiveWarehouse may be helpful. The initial search results make the project feel a little old and nonexistent. A little digging yielded a fairly active, well-documented project branch on GitHub: https://github.com/activewarehouse/activewarehouse-etl

Answer 2

Write your own. ETL is a very simple process; ruby ​​provides enough reflection support to handle this with simple code. ETL tools are not very useful here, just create point files to show data sources, streams and transformations.

I did the same in smalltalk for data conversion. There I used the glamor and Mondrian from the MOOSE reengineering tool to provide greater visibility.

Answer 3

Modulate, you want the Rails application and ETL to ask a question about the value of "HND" from the same place. To do this, configure the API.