Start a new topic

Performance improvement of data import

Hi


I would a appreciate if the performance of data import could be improved. I recently tried to import 300 million rows/20 GB file into a table. After over seven hours and only 11 % I stopped it and used DBeaver instead. It is about 10 times faster. I found IMPORT CSV file faster way giving an idea about the solution. Maybe, the source could be set up implicitly as a flat file table by according jdbc drivers (CsvJdbc, JDBC driver for CSV, cdata CSV JDBC Driver/XML JDBC Driver). However, I have no idea whether this would be any faster.


Kind regards


Thiemo


Hi Thiemo


Did you select the batch import option as indicated in the snapshot below?


Regards


Ulf



1 person likes this

Hi Ulf


Thanks for hinting me. I did not notice, it was switched of even though the loaded settings file had it on true. I fiddled with the batch size and now it rocks. :-)

However, I am concerned that loading settings will not respect all the values of the files namely


<BatchImport> and <Filename>

Maybe, my settings file is broken somehow.


Cheers


Thiemo

xml
Login or Signup to post a comment