Hello WebDesignerForum,

 

I've got a huge XML file. Currently it's about 56 GB and increasing size about 250 MB / week. I don't need all the elements in the XML file, and not all the elements contains the same information. How can I stream this file and check if a specific child exist and then insert it to my database, and if not then go for next element? I've tried searching for PHP scripts, VB.net and C# code, so i accept any help in these languages Thanks

I'm guessing it would be near impossible in php and cant comment on VB.net/C#

 

You could use node there are pre-made packages such as node big xml or xml object stream that can stream the file and parse it at the same time rather then storing it in memory then parsing it

I'm guessing it would be near impossible in php and cant comment on VB.net/C#

 

You could use node there are pre-made packages such as node big xml or xml object stream that can stream the file and parse it at the same time rather then storing it in memory then parsing it

 

In php i've heard about eventbased XML parsers which wont store the entire file in memory, but will look at each element at a time and then go through all the elements in the XML file.

Really? Why? I'd have thought it possible in PHP. I'd probably look at tying to split the XML file up into smaller chunks first though; which might be harder than the actual parsing of it.

I had a program splitting in into smaller parts over night, but now I'm left with 11500 files all with 750 elements each ~ 5-8 MB each. How would you do this?

Really? Why? I'd have thought it possible in PHP. I'd probably look at tying to split the XML file up into smaller chunks first though; which might be harder than the actual parsing of it.

TBH i have only worked a little bit with XML files/parsing in PHP but from the parses i have used before the XML file is always stored in memory and unless he has a system with 64GB+ it would not be impossible (this would be fixed by teh said event based parser).

 

Splitting up the files is such a pain in the ass, i had to do this when migrating our database from MsSQL to MySQL.

Without knowing what the xml looks like and what you want out of it it's hard to say but it's probably doable if the xml is well formed.

 

I've uploaded a sample of the XML to one of my test domains: You can view it here: http://view.qrdetector.dk/test.xml

 

It's a complete database of all the cars in Denmark which is why the main file is 56 GB. I need to extract some of the information like

<ns:RegistreringNummerNummer> and <ns:SynResultatStruktur> for each of the <ns:statistik>. Problem is that not all of them have <ns:SynResultatStruktur> and some of them have lots of information between this isnt relevant for my project.

 

A solution would be to store only one <ns:statistik> in memory, search through this and add the needed information to my database then remove from memory and repeat for next <ns:statistik> element

Splitting up XML is pretty easy if you know a bit of Xpath.

 

I'd have to ask though - why the hell do you have a 50gb+ sized XML file in the first place? That seems basically insane.

 

Its a complete databse of all the cars in Denmark

 

The link isn't working.

 

This is definitely doable with a bit of thought.

http://view.qrdetector.dk/test.xml

 

Sorry I added a space at the end of the link! Should work now