Convert Network Solutions CSV Product Exports to WooCommerce CSV Imports

This post will eventually detail everything you need to do to export Network Solutions products into WooCommerce formats for importing.

I wanted to start this post now even though it’s unfinished, so I can collect some feedback if this sort of thing is something more people are needing. There are no automated tools out there right now but I am working on automation in the form of PHP web scripts that will help convert the product exports from Network Solutions into CSV files that can be imported into WooCommerce.

So far I have scripts to help me parse out "simple" products, as well as "variable" products and their "variations". The way Network Solutions handles these data types is far different then how WooCommerce does, and they require a LOT of work to convert. I have something like 15 pages of notes on converting the various types, and 3 PHP scripts to parse the CSVs for some more advanced data manipulation, including the use of Network Solutions’ XML export as well.

Using my scripts you can automate the import of ALL your product images into Woo, as well as automate the creation of the Woo attribute and attribute_data columns for variable products and variations.

Using my notes and a spreadsheet program like Microsoft Excel you can rename columns, remove those you don’t need, recreate a proper product category structure in Woo styling, handle hidden or disabled products, create shipping classes from Netsol extra handling charges, and much more.

At this time these scripts and tools are not refined enough for the general public, but I can do this on a consulting basis if you would like to hire me to do this for your business. Feel free to contact me about converting your Netsol products to Woo imports.

In time I may release a more comprehensive automated scripting tool to do this for you.


More to come…

Please let me know if you are interested in these converting tools.

Mysqld.exe Process Taking Too Much RAM in WAMP?

If you find that mysqld.exe in Windows with WAMPSERVER is using a lot of RAM, this may help.

I have WAMPSERVER 2.4 installed with MySQL 5.6.12 as you can see here:



The mysqld.exe process is taking almost 500MB of RAM!



This has to do with a high default on the table_definition_cache variable of MySQL. Note that this variable may not have an entry in your my.cnf or my.ini file. Mine did not have an entry for this variable. However, when I checked the active variables using Navicat, it shows a value of 1400 as you can see here:



According to dev.mysql, the default value of MySQL 5.6 is 400. You can see here that it is actually a calculated value:

Notice that it says the value is based on a formula:

400 + (table_open_cache / 2)

So my table_open_cache is also not set in my.ini. My value happened to be at 2000. That answers that question. So why is table_open_cache so high? According to mysql, for MySQL version 5.6.12 or greater the default is 2000. There is the answer! I just happen to have installed MySQL 5.6.12 which set a table_open_cache of 2000 which, when using the calculation above set the table_definition_cache to 1400, which then used up about 500MB of RAM.

For my fix, I just added table_definition_cache = 400 to my.ini and restarted MySQL. Now it takes about 95MB of RAM.

My use of WAMP is fairly light, however, if you use yours heavily or keep a ton of data, you might fine tune these values to be more appropriate to your needs. Now I have a small table_definition_cache but still have a high table_open_cache. This may or may not cause issues in the future, but for now it’s just fine for a single user like me.

Hope that helps!

Repairing a Corrupt MYISAM Table with MySQL

It’s just one of those days! I began receiving emails like this:

Database error in vBulletin myversion:

Invalid SQL:
INSERT INTO searchcore_text (searchcoreid, keywordtext)
VALUES ( 1584876, ‘ SOME TEXT WAS HERE.’ )
ON DUPLICATE KEY UPDATE searchcoreid = VALUES(searchcoreid), keywordtext = VALUES(keywordtext);

MySQL Error : Table ‘./mydatabase/searchcore_text’ is marked as crashed and last (automatic?) repair failed
Error Number : 144
Request Date : Wednesday, January 29th 2014 @ 07:19:29 AM
Error Date : Wednesday, January 29th 2014 @ 07:19:29 AM
IP Address : xxxxx
Username : someuser
Classname : vB_Database
MySQL Version :

As you can see, no fun indeed. My vBulletin searchcore_text table corrupted. Because our forum is large with around 1000 people on at any given time, hundreds of these emails were coming in each day. I should have repaired this sooner but needed questions answered, and I was in the middle of a systemic server crash. But anyway.

The accepted advice from vBulletin is to run a repair on the table. I could not turn off MySQL because the forum needed to stay up, this ruled out using myisamchk as you are supposed to shut down mysqld for that tool. This means I had to use REPAIR TABLE from inside mysql itself. It is thought that this is slower, but oh well. I have a lot of data, but not THAT much (about 1.2 million rows in this table). So be it.

The first step in a MYISAM repair is make a backup. If your repair fails, it can cause worse damage, so back it up, and if the first repair doesn’t work, you can try another type. To get my backup, I couldn’t run mysqlhotcopy because it simply says the table is corrupt and quits. So I was left with the only option and that is to manually copy the 3 data files to a backup location. Any MYISAM table is made up of 3 files with extensions “frm”, “MYD”, and “MYI”.

To back them up, I first made a directory to store the backup, I put it in the home directory of the account with the corrupt table:

[bash light=”true”]cp searchcore_text.* /home/myaccount/safetybackup[/bash]

My files were about 1.5GB so it took a few seconds.

Next, I wanted to run the actual repair command from its own window session using the screen command. Since I am on Windows and use Putty to connect to my server, if my session is lost and Putty closes, I don’t want to lose the process or to be able to monitor it. So type this to create a screen for running the repair:

[bash light=”true”]screen –S repair[/bash]

Note that is a capital “S” and the word “repair” is just a name I’m choosing for the screen, it can be whatever name you want. Now, even if Putty crashes or times out, if I log back in to the server, this screen will still be there with any commands I’ve run still going.
With this screen attached (verify with ‘screen –ls’, also try ‘echo $STY’), I run the repair:

use your_forum_database;
repair table searchcore_text;

After that command I get a blank cursor just sitting there, the repair is now doing its thing.

repair table

I won’t go in to all the options of using screen, but at this point I opened a new Putty terminal and logged in to the server. You can also switch screens in the same terminal if you like.

In the second terminal window I have a fresh command line, and my mysql repair is happily going along in the other screen. You can see with “screen –ls” that your repair screen is attached.

Now I type this to see the process:


This gives me a small table where I see the “State” of the repair. It should say “Repair by sorting” for example. If it says anything else like using “keycache”, you may need to adjust your MySQL settings and try again. Keycache method could take forever, there are horror stories.

process list

In my case, I ran in to an issue where the /tmp folder filled up to capacity causing MySQL to just stop repairing. No errors, no messages, it just stops, you have to monitor your temp folder!

To fix that, I created a folder in the mysql data directory to use as a temp folder, set my.cnf variable, set “mysql” as the owner of the folder, and restarted MySQL. After that, I ran the repair again. Note that my /tmp folder was only 500MB, the repair easily needed over 1.2GB to run. With the new temp folder set, the repair operation took all of 5 minutes.

I verified the repaired table was good, then deleted my earlier backup. I also had to reset the mysql temp folder and restart the service again. I do that with this command:

/etc/init.d/mysqld restart

In a few seconds, it is restarted and rocking away! With the table repaired, there was a couple days of activity on the forum not indexed, so I did have to tell vBulletin to rebuild the search index. That itself is no small operation.