mardi 2 décembre 2014

How to handle big data update based on exel spreadsheet


Scenario



  1. Select-based data is exported from DB to Excel spreadsheets.

  2. Some manual work is done on the excel files (filling missing data).

  3. Changed data from excel files is updated back to DB.


Environment



  • Oracle DB

  • Original table is about 7 million rows

  • Can't use import tools in SQL Developer etc, must be a plain SQL script.

  • There are several excel files with couple hundred of thousand rows each (400-700 thousand)


Current solution


There is an UPDATE statement generated for each row of the excel file. It checks if data has been changed (the file has separate columns for old and new values). So from each file we get hundreds of thousands of UPDATE statements.


Question is...


This solution is fairly simple, but I wonder if there is a better way to do that?





Aucun commentaire:

Enregistrer un commentaire