Have Geek Brain

Importing a wikipedia database dump - kills navicat - anyone have any ideas?

Ok guys, I downloaded the wikipedia xml dump and its whopping 12GB of data: \ for one table and I wanted to import it to mysql databse on my localhost - however its a massive 12GB file and obviously navigators taking its sweet time when importing, or rather his hanging: (.

Is there a way to include this dump, or at least partially, you know in half.

Let me correct that its 21GB of data is not what it helps: \ - does anyone have any ideas to import such huge files into a MySQL database.

+1


a source to share


3 answers


Use command line instead, navicat is terrible for importing large files and will most likely take 20 times longer than using the CLI.



+2


a source


Take a look at the Sax parser , which allows you to read a chunk in chunks rather than just reading 12GB into memory. I'm not too sure how you could interact with mysql.



+1


a source


This is a pretty old question, FWIW .. refreshing with a new answer. i faced the same problems and hours for a single massive sql file to run can be risky, and running into any problems basically means you are starting over. what i did to lower the risk and get some performance through the CLI.

  • splits a massive SQL file into smaller, more manageable chunks, for example, "enwiki-20140811-page.sql" splits into files of about 75MB.

    split -l 75 enwiki-20140811-page.sql split_

    will produce a large number of files with "split_" prefix in the filename.

  • iterating over that list of files and importing one at a time ... a simple shell script as such.

    for f in $FILES
do
  echo "Processing $f file..."
  mysql -h $HOST -u $USER -p$PSWD $DB < $f
done

If it breaks down for some reason, you can easily return to where you left off.

Splitting the SQL file based on the number of lines prevents any large INSERT statements from breaking. However, if you lose too many rows, you can separate the DROP and CREATE statements at the beginning of the SQL. It's easy to install by opening the first few split files and resolving them.

0


a source






More articles:


All Articles