PHP WordPress Large background Importer Code

My wonderful client wanted me to add a .CSV importer that imports posts into a Custom Post Type(CPT) in large quantities of around 10k to 50k at a time.

What a standard WordPress site can handle

From my experience building plugins and importers for WordPress over the past 20 years, I have learned to expect trouble if you try to import anything more than 4k to 5k posts into WP at a time. The main requirement for the importer that I am building now is that it is able to import 20k posts in under 15 minutes. Not too difficult, but a challenge none-the-less because the average WP site will timeout after the first 4k posts on average. I also wanted to accomplish this goal without using any bloated PHP Que libraries or Worker Libraries, so I'll be developing it completely from scratch, just like a grown up!

Step One, Break Large File into Smaller Chunks

The first thing I did while developing my large WP file import feature, was to handle the issue of having a huge .csv file with 20 thousand posts saved in it. What do you do? What I did was wrote a PHP script that reads in that huge file and breaks it up into manageable chunks and saves each chunk to its own separate .csv file. Kind of simple really, this is the easy part!

I created a file named file_split.php to handle this portion of the importer's code. Here are the contents of that file:

<?php
$chunk_size = 2000;//size of ea. chunk file given in the no. of lines written to the file(excluding the first line with headings that will be at the top of ea. chunk file created).
$max_files = 0;//it will stop after writing this many files, set to 0 for no limmit
$u = $uploadpath;//"double_accounts-test.csv";
//read a file line by line:
echo "<p>Opening $u to spit it into chunks......</p>";
$file = fopen($u, "r") or exit("Unable to open $u!");
//extract the first line from the file:
$first_line = fgets($file);
//Output a line of the file until the end is reached:
$i=0;
$ii=0;
$iii=0;
$chunk=$first_line;
while(!feof($file) &&($iii < $max_files||$max_files==0)) {
$i++;
$ii++;
$line = fgets($file);
//echo "$i) $line<br>";
//add line to lines that get written to files:
$chunk .= $line;
//once we get to chunk_size lines, write a new file
if($ii == $chunk_size){
$iii++;
//echo "<h3>Chunk #$iii:</h3>";
//echo $chunk;
//create dynamic filenames:
$fn = ABSPATH."wp-content/plugins/YOUR-PLUGIN/temp/chunk$iii.csv";
//echo "Writing file $fn....<br>";
$fh = fopen($fn, 'w') or die("can't open $fn file to overwrite");
fwrite($fh, $chunk);
fclose($fh);
$m = "<span style='color:green'>Chunk Written to $fn</span><br>";
//reset $ii var back to zero:
$ii = 0;
//empty the $chunk var and add first line to top of file content($chunk)
$chunk=$first_line;
//echo "<hr><span style='text-align:center;color:orange;font-weight:bold;width:100%;margin-right:111px'>End File #$iii</span>$m<hr>";
echo $m;
//add the file name to the fnames array:
$fnames[] = $fn;
}//end if $ii = $chunk_size
}
fclose($file);
//if $ii is greater than 0 and less then chunk_size, then there was a left over chunk at the end, so write it to a file too:
if($ii > 0 && $ii < $chunk_size){
$noof_files = $iii+1;
$fn = ABSPATH."wp-content/plugins/YOUR-PLUGIN/temp/chunk$noof_files.csv";
echo "Left over chunk of $ii lines remains, so writing it to final file named $fn....<br>";
//echo "Writing Final file $fn....<br>";
$fh = fopen($fn, 'w') or die("can't open $fn file to write final chunk!");
fwrite($fh, $chunk);
fclose($fh);
echo "<h3 style='color:green'>FINAL Chunk Written to $fn! DONE chunking $u file! Total files created: $noof_files</h3>";
//add final file name to fnames array:
$fnames[] = $fn;
}else{
echo "Wow! We broke even with no left over chunk of less than $chunk_size lines, so finished writing files! Final file is $fn.<br>";
}

?>

Put the file_split.php file up for now. We will include it in another file later when we have the rest of our import feature code written. You can test file_split.php easy enough by

Leave a Reply

Your email address will not be published. Required fields are marked *