Jump to content

Speed up loading data table with many images


coolshrimp

Recommended Posts

ok saying i want to have thumbnails and images auto resized and saved to a folder.

any ready made scripts you know of ready to go?

i have a CSV (array) list of images to load and there files names to match.

if script can auto create thumbs and also auto delete when removed from array.

now would i have to manually run this file every time i want to update it or is there some way to make it automatic and can it compare image if link changes?

Edited by coolshrimp
Link to comment
Share on other sites

When page loads or when script for thumbnails runs, get a list of all files in folder and store in array, get list of images from excel and store in another array, remove from folder array images that exist in excel array, after looping through all you should end up with array of unused images, run loop to remove these images from medium and thumbnail folders, then run create thumbnail script to create new medium and thumbnail images that do not exist.

Link to comment
Share on other sites

When page loads or when script for thumbnails runs, get a list of all files in folder and store in array, get list of images from excel and store in another array, remove from folder array images that exist in excel array, after looping through all you should end up with array of unused images, run loop to remove these images from medium and thumbnail folders, then run create thumbnail script to create new medium and thumbnail images that do not exist.

you have Skype? if you wouldn't mind helping me out a bit.

Link to comment
Share on other sites

When page loads or when script for thumbnails runs, get a list of all files in folder and store in array, get list of images from excel and store in another array, remove from folder array images that exist in excel array, after looping through all you should end up with array of unused images, run loop to remove these images from medium and thumbnail folders, then run create thumbnail script to create new medium and thumbnail images that do not exist.

 

 

 

this on the right track?

 

 

<?php
//CSV Reader Scripts
$FileLocaton = $_SERVER['DOCUMENT_ROOT'] . '/php/CSVReader.php';
if (file_exists($FileLocaton)) {
    //Automatic Path
    include $FileLocaton;
} else {
   //Backup Maual Path
 	include 'php/CSVReader.php';
}

//Set CSV Location
$CSV = 'csvlist.csv';

//CSV Converted To Array
$CSVArray = csv_to_array($CSV);



$count = 0;
foreach ($CSVArray as $row) {
	$count++;	
		if ($count > 1 && $row[1] !== "") {		
		
		//Get FileID & Link URL from CSV	
		$id = preg_replace("/[^a-zA-Z0-9]/", "", $row[1]);
		$file = $row[8];
		
		//set thumb max dimentions
		$maxw = 200;
		$maxh = 200;
		
		//Create Tumbs
		CreateThumb($file, $maxw, $maxh);
				
		//Save Tumbs
		imagejpeg($img, "images/thumbs/{$id}" );				
		
		}
}


function CreateThumb($file, $maxw, $maxh) {
 
$sImagePath = $file;
 
$iMaxWidth = (int)$maxw;
$iMaxHeight = (int)$maxh;
 
if ($iMaxWidth && $iMaxHeight) $sType = 'scale';
 
$img = NULL;
 
$sExtension = strtolower(end(explode('.', $sImagePath)));
if ($sExtension == 'jpg' || $sExtension == 'jpeg') {
 
    $img = @imagecreatefromjpeg($sImagePath)
        or die("Cannot create new JPEG image");
 
} else if ($sExtension == 'png') {
 
    $img = @imagecreatefrompng($sImagePath)
        or die("Cannot create new PNG image");
 
} else if ($sExtension == 'gif') {
 
    $img = @imagecreatefromgif($sImagePath)
        or die("Cannot create new GIF image");
 
}

if ($img) {
 
    $iOrigWidth = imagesx($img);
    $iOrigHeight = imagesy($img);
 
    if ($sType == 'scale') {
 
        // Get scale ratio
 
        $fScale = min($iMaxWidth/$iOrigWidth,
              $iMaxHeight/$iOrigHeight);
 
        if ($fScale < 1) {
 
            $iNewWidth = floor($fScale*$iOrigWidth);
            $iNewHeight = floor($fScale*$iOrigHeight);
 
            $tmpimg = imagecreatetruecolor($iNewWidth,
                               $iNewHeight);
 
            imagecopyresampled($tmpimg, $img, 0, 0, 0, 0,
            $iNewWidth, $iNewHeight, $iOrigWidth, $iOrigHeight);
 
            imagedestroy($img);
            $img = $tmpimg;
        }     
 
    } 
 
}

}
 
?>
Link to comment
Share on other sites

anyway around it thinking its automated?

I don't know, I didn't write their code to detect scripts so I don't know what it's checking for.

 

this on the right track?

That code isn't going to save anything, the CreateThumb function should either save the image or return the image resource so you can save it.
Link to comment
Share on other sites

iv got it working :)

now how can i show each echo in script as it runs and not all at once when php is finished.

Here is code that is working:
Reads from my CSV, check is image tumbs aready exist, if not then creates a small thumb from (Image URL) and saves with (ID) iv specified in a column, then creates a big thumb and saves the same way.

<?php
echo "Creating Thumbnails<br><br>";

//CSV Reader Scripts
$FileLocaton = $_SERVER['DOCUMENT_ROOT'] . '/php/CSVReader.php';
if (file_exists($FileLocaton)) {
    //Automatic Path
    include $FileLocaton;
} else {
   //Backup Maual Path
 	include 'php/CSVReader.php';
}

//Set CSV Location
$CSV = 'CSVlinks.csv';

//CSV Converted To Array
$CSVArray = csv_to_array($CSV);

$count = 0;
foreach ($CSVArray as $row) {
    $count++;    
        if ($count > 1 && $row[1] !== "") {        
        
        //Get FileID & Link URL from CSV    
        $id = preg_replace("/[^a-zA-Z0-9]/", "", $row[1]);
        $file = $row[8];
        $saveto = "images/thumbs/";
        $SaveAsSmall = $saveto . $id . ".jpg";
        $SaveAsLarge = $saveto . "large/" . $id . ".jpg";            
        
        
            Echo "Checking for Large Thumb ({$id}.jpg)<br>";
            if (!file_exists($SaveAsSmall)) {   
                //Create Tumbs Small And Save
                $maxw = 200;
                $maxh = 200;        
                CreateThumb($file, $maxw, $maxh, $SaveAsSmall);
                Echo "Created Small Thumb for ({$id})<br>";
            } else {            
                Echo "Found Small Thumb for ({$id})<br>";
            }
        
        
            Echo "Checking for Small Thumb ({$id}.jpg)<br>";
            if (!file_exists($SaveAsLarge)) {   
                //Create Tumbs Small And Save
                $maxw = 400;
                $maxh = 400;        
                CreateThumb($file, $maxw, $maxh, $SaveAsLarge);
                Echo "Created Large Thumb for ({$id})<br>";
            } else {            
                Echo "Found Large Thumb for ({$id})<br>";
            }
            
        echo "<br><br>";
    
        }
}


function CreateThumb($file, $maxw, $maxh, $id) {
 
$sImagePath = $file;
 
$iMaxWidth = (int)$maxw;
$iMaxHeight = (int)$maxh;
 
if ($iMaxWidth && $iMaxHeight) $sType = 'scale';
 
$img = NULL;
 
$sExtension = strtolower(end(explode('.', $sImagePath)));
if ($sExtension == 'jpg' || $sExtension == 'jpeg') {
 
    $img = @imagecreatefromjpeg($sImagePath)
        or die("Cannot create new JPEG image");
 
} else if ($sExtension == 'png') {
 
    $img = @imagecreatefrompng($sImagePath)
        or die("Cannot create new PNG image");
 
} else if ($sExtension == 'gif') {
 
    $img = @imagecreatefromgif($sImagePath)
        or die("Cannot create new GIF image");
 
}

if ($img) {
 
    $iOrigWidth = imagesx($img);
    $iOrigHeight = imagesy($img);
 
    if ($sType == 'scale') {
 
        // Get scale ratio
 
        $fScale = min($iMaxWidth/$iOrigWidth,
              $iMaxHeight/$iOrigHeight);
 
        if ($fScale < 1) {
 
            $iNewWidth = floor($fScale*$iOrigWidth);
            $iNewHeight = floor($fScale*$iOrigHeight);
 
            $tmpimg = imagecreatetruecolor($iNewWidth,
                               $iNewHeight);
 
            imagecopyresampled($tmpimg, $img, 0, 0, 0, 0,
            $iNewWidth, $iNewHeight, $iOrigWidth, $iOrigHeight);
 
            imagedestroy($img);
            $img = $tmpimg;
        }     
 
    }
    
     imagejpeg($img, $id);                
    echo "Saved as {$id}<br><br>";
}

}
?>
Edited by coolshrimp
Link to comment
Share on other sites

now how can i show each echo in script as it runs and not all at once when php is finished.

That's the browser doing that, it waits until the connection closes to render the page. You would have to rebuild that to use ajax if you want to see updates while it works, where you send one ajax request for each file then display the output.
Link to comment
Share on other sites

That's the browser doing that, it waits until the connection closes to render the page. You would have to rebuild that to use ajax if you want to see updates while it works, where you send one ajax request for each file then display the output.

 

 

ok ill have to lear how this whole ajax thing works.

 

as for my script above see any issues? it seems to work :)

Link to comment
Share on other sites

This compare csv listing to image folder images (medium or thumbnail as they should have matching files just in different folders) content depending on extension (some folders have blank index.html) and store difference in a removal array which then removes these images (you will have manipulate path to remove images from medium and thumbnail folder), new images in csv will use your current script to retrieve image file externally and create thumbnail and medium sized images and place inn respective folder.

 

Just example, test this in demo image folder

<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <meta name="viewport" id="viewport" content="target-densitydpi=high-dpi,initial-scale=1.0,user-scalable=no" />
        <title>Document Title</title>
        <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
        <script  type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js"></script>
        <script type="text/javascript">

        </script>

        <style>


        </style>



    </head>
    <body>
        <table><tr><td>
                    <?php
                    $datafile = "../images/removetest/images.csv";
                    $csv = array_map('str_getcsv', file($datafile));

                    $csvImagesArray = [];
                    foreach ($csv as $row) {
                        $csvImagesArray[] = $row [0]; ///retieve image filename and path
                    }
                    unset($csvImagesArray[0]); //unset header row
                    $csvImagesArray = array_values($csvImagesArray); // clear unset array header from array

                    foreach ($csvImagesArray as $row) {

                        echo $row . '<br>';
                    }
                    ?>
                </td><td>
                    <?php
                    $allowed_filetypes = array('jpg', ' jpe', 'jpeg', 'png', 'gif');
                    $imgdir = '../images/removetest/';
                    $fullPathFilename_Array = glob($imgdir . "*.*");

                    $validFiles = [];

                    foreach ($fullPathFilename_Array as $fullPathFilename) {

                        $imageFileType = strtolower(pathinfo($fullPathFilename, PATHINFO_EXTENSION));  //retrieve extension only

                        if (in_array($imageFileType, $allowed_filetypes)) {
                            $validFiles[] = $fullPathFilename;
                            echo $fullPathFilename . '<br>';
                        }
                    }
                    ?>
                </td></tr></table>
        <?php
        $result = array_diff($validFiles, $csvImagesArray);

        foreach ($result as $remove) {
            unlink($remove);
        }


        print_r($result);
        ?>
    </body>
</html>
Link to comment
Share on other sites

on my script above in the function "CreateThumb"

 $img = @imagecreatefromjpeg($sImagePath)
        or die("Cannot create new JPEG image");

if it cant create image it will Die and echo error message.
how can i make it end the function but continue the script.

Die seems to stop whole script on error.
i want it to continue onto the next image.

Link to comment
Share on other sites

Thanks so far for your guys help.
I'm creating a script for all to use that will compare a list of files to directory of files and it will create or delete thumbs.

will post soon will need suggestions on how to run this each time person visits page without making them wait if possible.

Link to comment
Share on other sites

look over let me know if you see any possible issues. it seems to all work.
Now how can i run this script without affecting users page load?
will i just have to manually run it each time i make a change? or would a cron job be in order?

if using cron would i do:

curl http://sale.coolshrimpmodz.com/php/thumbnail/Thumb.php
or
wget http://sale.coolshrimpmodz.com/php/thumbnail/Thumb.php

THE SCRIPT


Thumb.php

<?php
/**
 *
 * @Created By Coolshrimp Modz
 */

/*error_reporting(E_ALL);
ini_set('display_errors', 1);*/

include '../logConsole.php';

//CSV Reader Scripts
$FileLocaton = $_SERVER['DOCUMENT_ROOT'] . '/php/CSVReader.php';
if (file_exists($FileLocaton)) {
    //Automatic Path
    include $FileLocaton;
} else {
   //Backup Maual Path
 	include '../CSVReader.php';
}

//Set CSV Location
$CSV = 'myfile.csv';

//CSV Converted To Array
$CSVArray = csv_to_array($CSV); //Get List of Images from CSV File

$logMessage = "";

// Set Folders To Save Images
$saveLocationSmall = '../../images/thumbs/'; //Small Thumbs
$saveLocationLarge = '../../images/thumbs/large/'; //Large Thumbs	

//Delete Old Tumbnails
include 'RemoveOLD.php';	

//Create New Thumbnails
include 'Create.php';
		
//Send Results to browser console
logConsole($logMessage);

?>

RemoveOLD.php

<?php
/**
 *
 * @Created By Coolshrimp Modz
 */


$filesList = array();

//Scan Small Tumbs Directory For Files with .jpg extention
$smallfilesArray = glob($saveLocationSmall . "*.jpg", GLOB_BRACE);

//Scan Large Tumbs Directory For Files with .jpg extention
$largefilesArray = glob($saveLocationLarge . "*.jpg", GLOB_BRACE);


//Get list of files from Array to be compared against
$count = 0;
foreach ($CSVArray as $row) {
	$count++;	
		if ($count > 1 && $row[1] !== "") {			
		//Get FileID & Link URL from CSV	
		$id = preg_replace("/[^a-zA-Z0-9]/", "", $row[1]);				
		$fileNameSmall = $saveLocationSmall . $id . ".jpg";
		$fileNameLarge = $saveLocationLarge . $id . ".jpg";
		$filesListSmall[] = $fileNameSmall;
		$filesListLarge[] = $fileNameLarge;		
		}
}


//Compare Arrays for diffrences
$resultSmall = array_diff($smallfilesArray, $filesListSmall);
$resultLarge = array_diff($largefilesArray , $filesListLarge);

//List Of Small Files Not On List To Be Removed
$count = 0;
foreach ($resultSmall as $row) {
	$count++;	
		if ($count > 1) {			
			//Get File Name
			$logMessage .= "Small Thumb To Be Removed  " . $row . "\\n";		
			
			//Remove The Images			
			if(file_exists($row)) { 
 				unlink($row); //delete file
 				
 				if(file_exists($row)) {
 					$logMessage .= "Small Thumb Could Not Be Deleted  " . $row . "\\n";
 				} else {
 					$logMessage .= "Small Thumb Deleted " . $row . "\\n";
 				}
 				
			}
		}
}

//List Of Large Files Not On List To Be Removed
$count = 0;
foreach ($resultLarge as $row) {
	$count++;	
		if ($count > 1) {			
			//Get File Name
			$logMessage .= "Large Thumb To Be Removed  " . $row . "\\n";		
			
			//Remove The Images
			if(file_exists($row)) { 
 				unlink($row); //delete file
 				
 				if(file_exists($row)) {
 					$logMessage .= "Large Thumb Could Not Be Deleted  " . $row . "\\n";
 				} else {
 					$logMessage .= "Large Thumb Deleted " . $row . "\\n";
 				}
 				
			}
		
		}
} 
?>

Create.php

<?php

/**
 *
 * @Created By Coolshrimp Modz
 */


include 'CreateThumbSave.php';

$count = 0;
foreach ($CSVArray as $row) {
	$count++;	
		if ($count > 1 && $row[1] !== "") {	
		
		//Get FileID & Link URL from CSV	
		$id = preg_replace("/[^a-zA-Z0-9]/", "", $row[1]);
		$file = $row[8];
			
			$SaveAsSmall = $saveLocationSmall . $id . ".jpg";
			$SaveAsLarge = $saveLocationLarge . $id . ".jpg";		
		
			if (!file_exists($SaveAsSmall)) {   
				$logMessage .= "Missing Small Thumb for " . $id . "\\n";
				//Create Tumbs Small And Save
				$maxw = 200;
				$maxh = 200;		
				$logMessage .= CreateThumb($file, $maxw, $maxh, $SaveAsSmall);				
			} else {			
				$logMessage .= "Found Small Thumb for " . $id . "\\n";
			}
			
			if (!file_exists($SaveAsLarge)) {   
				$logMessage .= "Missing Large Thumb for " . $id . "\\n";
				//Create Tumbs Large And Save
				$maxw = 400;
				$maxh = 400;		
				$logMessage .= CreateThumb($file, $maxw, $maxh, $SaveAsLarge);				
			} else {			
				$logMessage .= "Found Large Thumb for " . $id . "\\n";
			}			

	
		}
}

?>

CreateTumbsSave.php

<?php
/**
 *
 * @Created By Coolshrimp Modz
 */

function CreateThumb($file, $maxw, $maxh, $id) {
 
$sImagePath = $file;
 
$iMaxWidth = (int)$maxw;
$iMaxHeight = (int)$maxh;
 
if ($iMaxWidth && $iMaxHeight) $sType = 'scale';
 
$img = NULL;
 
$sExtension = strtolower(end(explode('.', $sImagePath)));
if ($sExtension == 'jpg' || $sExtension == 'jpeg') {
 
    if (!$img = @imagecreatefromjpeg($sImagePath)) {	
 		return "Cannot create new JPEG image for  " . $id . "\\n";
	}        
 
} else if ($sExtension == 'png') {
 
 	if (!$img = @imagecreatefrompng($sImagePath)) {	
 		return "Cannot create new PNG image for  " . $id . "\\n";
	} 
   
} else if ($sExtension == 'gif') {
 	
 	if (!$img = @imagecreatefromgif($sImagePath)) {	
 		return "Cannot create new GIF image for  " . $id . "\\n";
	} 
  
} else {
	
	if (!$img = @imagecreatefromjpeg($sImagePath)) {	
 		return "Cannot create new image for  " . $id . "\\n";
	} 

}

if ($img) {
 
    $iOrigWidth = imagesx($img);
    $iOrigHeight = imagesy($img);
 
    if ($sType == 'scale') {
 
        // Get scale ratio
 
        $fScale = min($iMaxWidth/$iOrigWidth,
              $iMaxHeight/$iOrigHeight);
 
        if ($fScale < 1) {
 
            $iNewWidth = floor($fScale*$iOrigWidth);
            $iNewHeight = floor($fScale*$iOrigHeight);
 
            $tmpimg = imagecreatetruecolor($iNewWidth,
                               $iNewHeight);
 
            imagecopyresampled($tmpimg, $img, 0, 0, 0, 0,
            $iNewWidth, $iNewHeight, $iOrigWidth, $iOrigHeight);
 
            imagedestroy($img);
            $img = $tmpimg;
        }     
 
    }
    
 	imagejpeg($img, $id);
	return "Thumb Created And Saved To " . $id . "\\n";

}

}

?>

 

Link to comment
Share on other sites

can i just do this use ajax to run script dosent seem to affect page at all and user will not know its running.
it also appends to body the console log once it is done so i will know what went on. is this a proper use or any issues with this?

<script type="application/javascript">        

function loadimages(){
 	$.ajax({
 		type: "POST",
 		url:"php/thumbnail/Thumb.php",
 		success: function(html){ 			
 			$('body').append(html);
 		}
 	});
}

window.onload = function() { setTimeout(loadimages, 2000); };

</script>

Seems to work. load page and watch console log for change:

http://sale.coolshrimpmodz.com/index.html
Edited by coolshrimp
Link to comment
Share on other sites

will i just have to manually run it each time i make a change? or would a cron job be in order?

Whichever you'd rather do.

 

You can call php directly instead of curl or wget, and pass PHP the filename to execute.

 

can i just do this use ajax to run script dosent seem to affect page at all and user will not know its running.

It's going to run in the background, so any images that don't exist when they load the page for the first time aren't going to automatically show up when that PHP script runs.
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...