Merge and extract product data from numerous SQL tables to Hadoop key-value store with Scala / Python -


Advice on the following work: Combine and remove product data from several SQL tables. These tables include Product IDs and various features of this product in Records. As a result the data should be stored in an HDFS table, where a given record shows all the properties related to a given product. The result may be the variable number of properties in the record, for example:

  Input SQL record: Table 1 Rick 1: | ProdId | X11 | X12 | X13 | ... | X1n | Rick 2: | Prodod | X21 | X22 X23 ... | X2n | .... Rick M. | Prodod | XM1 | XM2 XM3 | ... | XMN | Table 2 Rick 1: | Prodied | Y11 Y12 Y13 ... | Y1 Rick 2: | Prodied | Y21 Y22 Y23 ... | Y2k | ... Rick L. | Prodod | YL1 | YL2 YL3 | ... | XLK | .... .... .... Table Jade Rick 1: | Prodied | Z11 | Z12 | Z13 | ... | Z1Q | Rick 2: | Prodied | Z21 | Z22 | Z23 | ... | Z2Q | .... Rick P: | Prodod | ZL1 | ZL2 | ZL3 | ... | ZPQ | Output HDFS Table: Rick 1: | ProdID | X11 | X12 | X13 | ... | X1n | Y11 Y12 Y13 ... | Y1k | Z11 | Z12 | Z13 | ... | Z1Q | Rick 2: | Prodod | X21 | X22 X23 ... | X 2 n | Y21 Y22 Y23 ... | Y2k | Z21 | Z22 | Z23 | ... | Z2Q | .... RC R: | ProdId | XI1 | XI2 | XI3 | ... | XMN | YJ1 | YJ2 YJ3 ... | XLK | ZF1 | ZF2 | ZF3 | ... | ZFQ |  

The result should be field names and stored in HDFS so that the written jobs in Scala / Java / Python can be reduced.

i am new in handop and it may be wrong to use the honeycomb for this work. I think it is a good idea to use the map-less, yet I do not know how to import the input SQL data To do this, punctuate the other with scooped designated areas. Is there any way to export the SQL Table to HBEES, which will automatically reconstruct the field names? Any other important value for this work that supports designated areas is good store? Ideally, the framework should be detected which will allow this work in Scala or Python.

AFAIK, this task is usually done by the tool, its purpose is to expand it with Scala It is entirely possible, because it has been written in Java.


Comments