Geeks: Importing products in Magento (1.4+) #1

First things first

We’ll be working with a CSV file as our data source for product data, as this seems to be the most commonly used. If you’re following the article step by step, you’ll want to download the sample file here: starwars.csv and save it somewhere that the code we’re building is able to access it again.

There will be mostly code examples in this articles, and not much talk about the “inner workings”. Most people including myself, like to learn by doing, and not by reading so I figured we would try this approach first and see what happens. I tried commenting most of a bit.

Product data is random data from shop.starwars.com

This is a standard UTF-8, comma-seperated CSV file like any you would receive from a client, with the following fields defined:

Note: The CSV file you download does not contain the “sku”, “name”, … headers – these are just to illustrate what each field is.

Creating your skeleton file

As with most scripts dealing with anything Magento, we’re going to start out with a very basic PHP script that simply includes app/Mage.php and sets the current store to admin. If we don’t do this, saving and even loading products usually throws an exception.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?php
# path to app/Mage.php
# replace with whatever you need to, so it matches your site
require "/var/www/theforce.dk/public_html/magento/1.5.1/app/Mage.php";
 
if (class_exists('Mage'))
{
        # this is important, otherwise you'll get errors when trying to
        # work with products
	Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
 
	    echo Mage::getVersion();
	// more code
}

Save the script, upload it and run. You should see it echoing out the version number of your Magento installation. If all that went well, you’re ready to read on. But wait, if you haven’t downloaded the example CSV file, I suggest you do so before continuing.

Parsing the CSV product data

There’s alot of ways and smart ways and functions (str_getcsv) to parse CSV data, but for this simple example we’re just going to use the good old fgetcsv.

Anyway, let’s see some code that’ll parse the CSV file and output all the elements we get back.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

<?php
# path to app/Mage.php
# replace with whatever you need to, so it matches your site
require "/var/www/theforce.dk/public_html/magento/1.5.1/app/Mage.php";
 
if (class_exists('Mage'))
{
	# this is important, otherwise you'll get errors when trying to
	# work with
	Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
 
	# set input file
	$file = './starwars.csv';
 
	# open file and parse it
	if (($handle = fopen($file, 'r')) !== false)
	{
		while (($data = fgetcsv($handle, 1000, ',')) !== false)
		{
			//Array
			//(
			//    [0] => 1308390
			//    [1] => Darth Vader Toaster
			//    [2] => Burns Darth Vader image into toast
			//    [3] => 10
			//    [4] => 54.99
			//    [5] => Collectibles
			//)
 
			print_r($data); # produces the above array
		}
	}
}

Run the above script and see if you get the same output as the comment says. If so, you’re ready to move on again. The code below should be pretty self explanatory, as most of Magento’s namings make sense.. like name, description, tax_class_id and so forth.

Using our CSV data for something useful

Now we go a bit deeper, and try to use all this data we’ve extracted. We’re trying to import all the data into Magento, as simple products. I won’t be covering configurable products and attaching simple products in this article – we’ll save that for part 2.

Take a look at the following code and try to understand what’s going on – I’ve made minor comments in the the code, and tried to make it as obvious and simple as i possibly could.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

<?php
# path to app/Mage.php
# replace with whatever you need to, so it matches your site
require "/var/www/theforce.dk/public_html/magento/1.5.1/app/Mage.php";
 
if (class_exists('Mage'))
{
	# this is important, otherwise you'll get errors when trying to
	# work with
	Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
 
	# product model
	$model = Mage::getModel('catalog/product');
 
	# default attribute set id (usually "default")
	$attribute_set = Mage::getModel('eav/entity_type')
		->loadByCode('catalog_product')
		->getDefaultAttributeSetId();
 
	# locking the indexer while import speeds up the process by atleast 50%
	# as Magento doesn't have to reindex the product data on _every_ save()
	Mage::getSingleton('index/indexer')->lockIndexer();
 
	# set input file
	$file = './starwars.csv';
 
	# open file and parse it
	if (($handle = fopen($file, 'r')) !== false)
	{
		while (($data = fgetcsv($handle, 1000, ',')) !== false)
		{
			//Array
			//(
			//    [0] => 1308390
			//    [1] => Darth Vader Toaster
			//    [2] => Burns Darth Vader image into toast
			//    [3] => 10
			//    [4] => 54.99
			//    [5] => Collectibles
			//)
 
			# print_r($data); produces the above array
			
			# weather product is in stock or not
			$isinstock = ($data[3] > 0) ? 1 : 0;
 
			# product data
			$productdata = array
			(
				'type_id' 			=> 'simple',
				'attribute_set_id' 	=> $attribute_set,
				'sku' 				=> $data[0],
				'name' 				=> $data[1],
				'description'  		=> $data[2],
 
				# the requirement of this can be turned off in Magento
				'short_description' => 'short description',
 
				'price' 			=> $data[4],
				'category_ids' 		=> array('2'),
				'tax_class_id' 		=> 2,
				'status' 			=> Mage_Catalog_Model_product_Status::STATUS_ENABLED,
				'visibility' 		=> Mage_Catalog_Model_Product_Visibility::VISIBILITY_BOTH,
				'website_ids' 		=> array(1),
				'weight' 		=> 1,
				'created_at' 		=> strtotime(now()),
 
				# product stock data
				'stock_data' 		=> array
				(
					'qty' 		  => $data[3],
					'is_in_stock' => $isinstock,
				)
			);
 
			# remember to use try/catch statements as imports will fail
			# from time to time
			try
			{
				echo "Saving {$data[1]}\n";
 
				# set product data to the array we specified above
				# note:
				# it's possible to use eg.
				# $model->setName('Master Yoda');
				# $model->setSku('123456');
				# etc. instead, but I find writing all the information
				# in an array looks nicer.
				$model->setData($productdata);
				$model->save();
			}
			catch (Mage_Core_Exception $e)
			{
				print_r($e);
				print_r($product);
			}
		}
	}
 
	# remember to unlock it again
	Mage::getSingleton('index/indexer')->unlockIndexer();
}

Sorry about some of the indenting – I copied parts to TextMate and back to Coda and now none of them seems to want to fix their tabs/spaces mess.

A note about the tax_class_id property on the product. I find that this number changes from store to store, sometimes it’s 1, sometimes it’s 2 – it all depends on how tax is set up on your shop. On a default Magento installation it’s usually 2. I’ve not yet found any Mage::* calls that’ll give me this number back, so I usually inspect the dropdown when editing a product in the administration, and get the value ID of the Taxable Goods entry. If you know of anything related to this, please post a comment, thanks!

Update: It appears that installing the module Crius ConfigDanish which we do with basicly all our clients, the ID changes to 1 as it’s changing some tax settings.

If you’re running the script from a terminal like I am, then you should see something similar to this:

And that means, when you go and check the product list in your Magento administration, you (hopefully) see your newly imported products:

If that’s the case, good job – simple isn’t it? If not, don’t sweat. Read the article again, or post a comment if you run out of ideas.

Finally, reindex the entire store – that means, eav structure, catalog search and so forth as these have not been run because we lock the indexers. It’s probably a good idea to clear the cache (but you don’t have cache enabled in a dev. environment – do you?!) also.

Regarding the two calls to

1
2

Mage::getSingleton('index/indexer')->lockIndexer();
Mage::getSingleton('index/indexer')->unlockIndexer();

I found these digging around in the Indexer class. These are actually not used anywhere in the Magento core, but by turning off the indexer you speed up the import by atleast 50%, because you’re telling Magento not to run an index each time you save a product. Nice “hidden” feature, at least I didn’t know about until recently when I tried to run a small benchmark test.

Imported 100 products in 60.9851369858 seconds, WITHOUT indexing fix
Imported 100 products in 29.3310470581 seconds, WITH indexing fix
Imported 50 products in 27.5145599842 seconds, WITHOUT indexing fix
Imported 50 products in 11.2406229973 seconds, WITH indexing fix

Over 50% speed increase. That’s alot if you know how slow importing can get in Magento.

Note: It is also possible to lock the indexer in the Magento administration “somewhere”, but as import scripts usually get changed from time to time it gets annoying not remembering to do this, especially if you’re working with alot of products.

In part 2 I’ll be covering some more advanced subjects

• Importing configurable products and attaching simple products (variants)
• Importing and attaching images (single/multiple) to products
• Importing and attaching categories to products
• .. more

Feel free to comment for help/advice/tweaks/fixes/.. to above code.

Stay tuned.