Before I start, if you have any doubts or you are unaware about character sets (i.e. you are not familiar with words like utf-8, unicode etc), I would recommend you to read Joel's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
The problem statement of this article is How to handle foreign or accented characters in html screen scraping. We encountered it while working on our website information project Quarkbase.comConfiguration & Tools we will be using are rails - 2.1.2, scrapi - 1.2.0, hpricot - 0.8.1, curb - 0.4.4.0. The example website that we will try to scrape is http://196m.cn
At Quarkbase, when we aggregate information from a website, scraping information (having non-english characters) doesn't always work. For example, when we get description for http://196m.cn from its HTML page, the extracted information is the first line after 'overview' in screenshot below :
Now, its not like only I don't understand it, no one can understand it :P, because they are not valid characters at all. So, we need to fix this issue and fetch readable utf-8 data while scraping.
The first step will be to fetch the html document using any of the ruby's URL library (net/http, open-uri or curb). We will be using curb here.
1 2 |
curl_object = Curl::Easy.perform("http://196m.cn") |
curl_object.content, i see a lot of "?", and other garbage characters. if I try to parse it using scrapi or hpricot, i get some crap, which makes no sense and is unreadable to all.Before scraping
curl_object using Hpricot or Scrapi, I should tell you that Hpricot requires input in utf8-encoding and Scrapi has no such condition but it does not always work with other character encoding.
So, we should convert html data in curl_object to utf-8 encoding, for which we need current encoding of the document. One way to do this will be:
1 2 3 4 5 6 7 |
if meta = curl_object.content.match(/(<meta\s*([^>]*)http-equiv=['"]?content-type['"]?([^>]*))/i) if meta = meta[0].match(/charset=([\w-]*)/i) encoding = meta[1] end end => 'gb2312' |
1 2 3 |
require 'iconv' Iconv.conv('utf-8', encoding, curl_object.content) |
1 2 |
Iconv.conv('utf-8//IGNORE', encoding, curl_object.content) |
Cool! Now we have content in utf-8 format. Lets scrape it!
Using Hpricot:
1 2 3 |
Hpricot(Iconv.conv('utf-8//IGNORE', encoding, curl_object.content)).search("meta") => #<Hpricot::Elements[{emptyelem <meta content="text/html; charset=gb2312" http-equiv="Content-Type">}, {emptyelem <meta name="description" content="包括茶叶及茶饮料,茶食品,搜茶,茶民周刊,茶文化,音乐红茶馆,下午茶,茶道,茶叶选购指南及茶具陶瓷,茶业种植,加工,营销等频道及专题栏目,是以茶道茶艺茶文化为主题的综合性茶道网站">}, {emptyelem <meta name="keywords" content="茶,茶叶,茶网,茶文化,茶艺培训,茶道,tea,茶文化,茶业,红茶,绿茶,铁观音,茶饮,茶艺,茶餐厅,茶具,紫砂,龙井,碧螺春,乌龙茶,大红袍,普洱茶">}]> |
Using Scrapi:
1 2 3 4 5 6 7 8 9 |
Scraper.define do process "meta[name='description']", "r[]"=>:element result :r end.scrape(Iconv.conv('utf-8//IGNORE', encoding, curl_object.content)).to_s => "<meta name=\"description\" content=\"包括茶 å¶åŠèŒ¶é¥®æ– ™,茶食å“,æœè Œ¶,茶民&....... |
This is not the expected output. Scrapi comes with 2 parsers: tidy and html_parser. Scrapi by default uses tidy, so lets try using html_parser
1 2 3 4 5 6 |
t=Scraper.define do process "meta[name='description']", "r[]"=>:element result :r end.scrape(Iconv.conv('utf-8//IGNORE', encoding, curl_object.content), :parser=>:html_parser).to_s => "<meta name=\"description\" content=\"包括茶叶及茶饮料,茶食品,搜茶,茶民周刊,茶文化,音乐红茶馆,下午茶,茶道,茶叶选购指南及茶具陶瓷,茶业种植,加工,营销等频道及专题栏目,是以茶道茶艺茶文化为主题的综合性茶道网站\">" |
I’m so happy I stumble upon this blog! A lot of helpful info I really needed to know these stuff, I had a hard time with those foreign characters. Thank you so much!
Another very solid tutorial. Thank you. Not something I can use right now but I bookmarked for the future.
Matt Gold
This looks brittle. What if the encoding is defined in the HTTP header? With normal open() instead of curl you get the encoding for free: http://jasonseifer.com/2008/03/18/hpricot-and-utf-8
Hello, the type code really help me much. Thank you for the suggestion and code.
It is very useful, thanks.
Good Luck
A lot of helpful info I really needed to know these stuff, me too
Hello, the type code really help me much. Thank you for the suggestion and code.
excellent information. Quite helpful. I gonna share this with my communities.
He was not a favourite at the court of Lorenzo the Magnificent as Filippino Lippi and Botticelli were. Lorenzo liked those who would flatter him and do as they were bid, while Leonardo took his own way in everything and never said Cat Supplies what he did not mean.
I have been trying to look for a solution to the utf8 problem and glad I searched and found you. Now I can save myself a lot of time by not having to figure out myself.
Good article. Thank you so much to share!
Thanks for sharing this great article.. Like it! Pinetop Cabin Rentals
Another great set, I appreciate all the work you put into this site, helping out others with your fun and creative works.
Thank you so much to share!
This is very good information!
Its always good to learn tips like you share for blog posting. As I just started posting comments for blog and facing problem of lots of rejections. I think your suggestion would be helpful for me. I will let you know if its work for me too. Thank you for sharing this beautiful articles.
These facts are amazing . I was searching before last 5 weaks and ia din’t get the perfect answer. But after all i found from your site. thanks for posting such a interesting topic.
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you. Cleaners | Painters | Plumbers
Yet another problem with these changes. I am not able to access instance methods of my derived class from Scraper::Base from these blocks. I think these blocks are not “bind” to the local instance properly. Following code is an example from base.rb of scrapi and this doesn’t work.
The Acronym List is a searchable database of over 8 million acronyms, abbreviations and meanings. Covers: business, international, chat, organizations, common acronyms, computers, science, technology, government, telecommunications, and military acronyms.
Try the Burton Blunt or the Capita Scaremaster or a Rome cheap trick. those are all cheap and good for park. e-business
Most will cut them to size for you if you ask. They may charge a small fee. If they don’t know what they are doing usually you can use 1×4 oak boards and they will work fine. e-business
good tuto thanks !
Good website. I like the all pages and all comments. Thanks for all!!! Regards!
I was searching before last 5 weaks and ia din’t get the perfect answer. But after all i found from your site. thanks for posting such a interesting topic.
It’s a very exciting article. I like it so! i was very focused when i’m reading this. Thanks
I am new to scrapping but I think its quite an awesome technique.No doubt there are lot features available in different programming languages which we can use to scrap a web site very easily.
I honestly think that just using Rails has made me a better developer. I can’t say much about Coldfusion because I’ve never used it, but it’s good that you are considering the merits of each, best to use the tools that are right for the job. Speed Dating New York
Shop the latest styles juicy couture handbags, juicy couture tracksuit. Juicy Couture
Really great information and inspiration, both of which I like. Nice article with very strong content. I love reading it over again.
cool information – difficult but interessting!
autoradio navigation, car dvd gps navigation, sat navigation stereo, OEM Factory headunit for all car makes Higher quality car electronics from Qualir Car DVD Player
I like this article very much. Thanks for your share!
FashionStyleOnsale offer high quality Moncler Jackets at low price. Moncler Jackets on sale, shop more discount Moncler Vest, Moncler Coats at FashionStyleOnsale Moncler
FashionStyleOnsale offer high quality Moncler Jackets at low price. Moncler Jackets on sale, shop more discount Moncler Vest, Moncler Coats at FashionStyleOnsale Moncler
Merci pour cet article car l’integration des characters dans les codes html c’est pas evident.
cool information – difficult but interessting!
Funny, I bought google phone and I love google map. Problem is google map keep giving me wrong direction. Email is nice though, I wish i can check hotmail and yahoo too.
I had a possibility to buy a smartphone, but I didn`t, I`ve got accustomed to my ordinary phone. But I liked your article.
Thanks for the updates. Not sure I would’ve been able to do this without them. Cheers once again.
This is a wonderful blog
Great stuff here. Thanks
It is a pleasure to read your articles. I have got a lot of useful information from your site. I was totally unfamiliar with many of subjects, but now I feel pretty confident in it! Thanks for sharing this site.
I would like to thank you for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well.
I’m glad i found ur blog.Not everyone can provide information with proper flow. Good post. I am going to save the URL and will definitely visit again. Keep it up.
that is such a very good information in your blog i am learning many tools from here i am going to bookmark this page. regards lyrics team lyrics lovers.
It’s an interesting approach. I commonly see unexceptional views on the subject but yours it’s written in a pretty unusual fashion. Surely, I will revisit your website for additional info.
i´m so very happy, that i found this interesting post.
thank you!3
Nice work, even that not very simple
Thanks admin. Good site
Excellently written article, if only all bloggers offered the same content as you, the internet would be a much better place. Please keep it up!
mason0507 tk
Resources like the one you mentioned here will be very useful to me! I will post a link to this page on my blog. I am sure my visitors will find that very useful.
this is a great post and thank you for sharing this nice experience,and hope you can give another posts as soon as possible.
I don’t know how to say with you. So much useful information i need!Thanks alot!
I like it ! Thank you for this post, It was a great read which was extremely helpful.
Thanks for a great tutorial. I wouldn`t do it on my own.
Remember you will not always win. Some days, the most resourceful individual will taste defeat. But there is, in this case, always tomorrow – after you have done your best to achieve success today.
This is by far the simplest tutorial on this subject matter that I have come across. Great share.
When you say to ask Iconv to convert as much as it can does this mean that data will be lost?
This is the exact way that people expecting we can see most of the comments are true and towards the positive opinion..This is the exact way that people expecting we can see most of the comments are true and towards the positive opinion..
Thanks for the UTF-8 and html screen scraping in Ruby on Rails which is really useful for me !
Ruby on Rails is new to me. thanks for the info.
I never would have thought to do it this way. You saved me a ton of time. Thnaks
mason0507gawwr
Thank you for this wonderful illustration الجياش
Thank you for this wonderful illustration
Great blog and screen shots make it so much easier
thnx 4 sharing برامج الجياش
Although made famous by its boots, UGG Australia has extended their product line to include stylish yet elegant clothing for both adults and kids. UGG Australia has designed a vast collection of gloves, scarves, hats, and UGG coats that will wrap you in warmth against the cold winter months. When the weather heats up and warm breezes blow, you can now enjoy cute dresses, wraps and lightweight hats.Now you can accessorize your favorite outfit with not only UGG footwear but a fabulous bag too. You can travel with a tote, shop with a cross-body messenger or paint the town red with a chic clutch or wristlet. All UGG handbags are crafted with the same attention to detail that has made them the premier brand they are today.What started off as a way to keep surfers’ feet warm has grown to include: UGG boots, slippers, kids, clothing, bags and accessories.
It fits perfectly. That translates to it running a half size smaller.I have only worn these boots for one day, but for 12 hours (it was a long day), and they were very comfortable.I personally prefer to pull the furry flap UP instead of folded like in the photo. It reveals more of the grey which I like. With jeans just barely tucked in, it looks really chic despite the fact that it’s a weather boot.I also want to add that I am not a fan of the slouchy UGG boots. I avoided this brand because of that image, but these Adirondack boots are something else. They are well worth it.
Thank for the post. At the moment I do not need it but who knows what lies in the future.
Your blog is so nice.I am impressed with your vivid expression.I will bookmarked you…keep up the good work!!!!
Another very solid tutorial. Thank you. Not something I can use right now but I bookmarked for the future.
I will definitely follow it, thanks and keep up the good work.
I’m looking forward to reading more
UGG Australia is a brand that is all about luxury and comfort for everyday life.
Although made famous by its boots
Thanks for keeping us informed with all these news!
yes you can see this great and what you have to say.
for the baby to sit next to me in his career. If I could recover in a niche you can
Some media are owning to spend attention to this.
Well Done!I do wish u can publish drastically more posts later I even now have an extraordinary offer to research about silverlight.
yup I do wish u can publish drastically more posts later I even now have an extraordinary offer to research about silverlight.so keep on marriage and say all is well.
What are the single product this season is will enter almirah, and what is the most popular sheet is tasted beauties stars favorite?Sapphire blue successful open European nobility female gate, gossip girl under the lead lifted a stream of blue storm…Gossip: what kind of dressing girls have treasure?Together and see it!
I’m a new big user so need as much help as possible.the great
This warm degrees best feet accessories and is worthy of many women love most, except warm outside, boots peculiar on the legs of the modification, also is other hard to deserve to comprehend.Skirt or Legging collocation are together, in winter seems like “necessary”.But sometimes this kind of collocation see more will aesthetical fatigue, true want to shout a voice in the streets: isn’t big boots besides collocation of black silk, Legging and no better choice?
What are the single product this season is will enter almirah, and what is the most popular sheet is tasted beauties stars favorite?Sapphire blue successful open European nobility female gate, gossip girl under the lead lifted a stream of blue storm…Gossip: what kind of dressing girls have treasure?Together and see it!
I think it opens the website is very nice and I like to have web sites of mine, but you so beautiful I do not how good web design and people who collect my website could you help me friends
This is a very good post, I learn very much, I will invited my friend to vist it. our site is also very good, we have [url=http://www.coachoutletus.net]coach outlet[/url]t, [url=http://www.coachoutletus.net]coach handbags[/url], [url=http://www.coachoutletus.net]coach bags[/url], [url=http://www.coachoutletus.net]coach outlet store[/url],welcome to vist.
This is a very good post, I learn very much, I will invited my friend to vist it. our site is also very good, we have coach outlet, coach handbags, coach bags, coach outlet store,welcome to vist.
good poset !I do wish u can publish more posts later I even now have an extraordinary offer to research about silverlight. Lunettes Ray Ban
Great job we like it . cadeaux naissance bébé
Hi all,
Thanks for help to understand characters of computers ! Because it’s for us really easy to find something on this site to improve ours about the financial advises and Forex. Thank you very much !
Thanks for help to understand characters of computers ! Because it’s for us really easy to find something on this site to improve ours about the financial advises and Forex . Thanks all much !
Thanks for help to understand characters of computers ! Because it’s for us really easy to find something on this site to improve ours about the financial advises and Forex . Thanks all much !
This warm degrees best feet accessories and is worthy of many women love most, except warm outside, boots peculiar on the legs of the modification, also is other hard to deserve to comprehend.
An interesting discussion is worth comment. I think that you should write more on this topic, it might not be a taboo subject but generally people are not enough to speak on such topics. To the next. Cheers for xbox 360..
Hello, the type code really help me much. Thank you for the suggestion and code.
The article is quite interesting and it gave me very reliable and useful information. I like to read such valuable articles as it increase my knowledge of iniSip
A nice tip,very thanks for your tip.
Great tutorial and very well written. Thanks for sharing ;)
It’s good to see th hanx now i have the link which i was looking for my research.
I am very pleased that I was able to find such a great resource for gaining a deeper knowledge on the topic. It is always very important to study various opinions, evaluate them and get hold of a best answer for any question someone may face. That is exactly what I got here, surprisingly, free of charge. Thanks a lot!
The problem statement of this article
This is a smart tutorial… thanks for share
i really nice tutorial, yet more help for my printing site, without these online i’d really struggle to build my sites.
i agree, it’s good for my range cookers site too.
unbelievable, i’ve got a printing site too which i used this on! Cheers!
i really nice tutorial, yet more help for my printing site, without these online i’d really struggle to build my sites.
Top grade UGG boots;The best Replica UGG boots
Although in the production of wool boots and wool products, and in blessing Ken bridge built up his own UGG stores.
here are various models on the market the product more and more
This code will be very handy to have when the time comes to use it.
xiaoru
xiaoru
Hi, for any thesis help, dissertation help and Online Tutoring that you need in Engineering subjects that you need, you can visit our website at http://www.helpwiththesis.com . We provide the above services at a minimal cost.
Solid tutorial and explanatory. Congratulations. It was exactly what I was looking for. Thank you.
Thanks a lot for this information i think that what you have write is true and a
lot of user can be read more about this tread
downloadyoutubevideos
Thanks a lot for this information i think that what you have write is true and a
lot of user can be read more about this tread
Thanks a lot for this information i think that what you have write is true and a lot of user can be read more about this tread red riding hood download limitless download watch limitless online watch the lincoln lawyer online
Another great set, I appreciate all the work you put into this site, helping out others with your fun and creative works. Arabic Alphabet arapska slova
Though a bit over my head the information you have provided is truly valued, thank you for the post. But then again I guess if I read a few more things here I’ll be able to understand it a bit better, thanks again.
This is a very good post, I learn very much, I will invited my friend to vist it. our site is also very good…...... cnc spindle repair
The quarkruby is a blog by ritesh arora and nakul aggarwal, where they frequently share their experience & findings about ruby ror.Thanks for sharing the informative post. Regards. david77 – bathing suits
thanks i have a few ebooks adressing this subject. i’d be happy to share.
Really great information and inspiration, both of which I like. Nice article with very strong content. I love reading it over again. spindle repair
A lot of helpful info I really needed to know these stuff, me too. masters in education
You can fully relax yourself in NYC to Canada Tours.
When you say to ask Iconv to convert as much as it can does this mean that data will be lost? masters in public health
I agree with your point of view on this topic. I think this needs to be discussed more among the general population. professional photo printer
I think there is a better way to achieve what you achieved. It requires less steps and less semantics. profit jackpot
Hi, I think this is a very nice post. It will help me and everyone much. Thanks.Mattresses
This is the exact way that people expecting we can see most of the comments are true and towards the positive opinion..This is the exact way that people expecting we can see most of the comments are true and towards the positive opinion.. hotels in palm springs
These are some of the best tutorials I have come across. thanks for putting this out.
He was not a favourite at the court of Lorenzo the Magnificent as Filippino Lippi and Botticelli were. Lorenzo liked those who would flatter him and do as they were bid, while Leonardo took his own way in everything and never said Cat Supplies what he did not mean.
the code is somewhat complex. Any idea on how to simplify it? Thanks
Wow , that is pretty bold. She has got like 5 different looks going on there. high heels, fur, big glasses, it looks like she just grabbed everything laying around and put it on.
Great blog. It has been helpful in the development of our new site
She has got like 5 different looks going on there,,Great blog,I think this is a very nice post
Thanks wrote on this topic. Thought no one will bother. Though I don’t agree with everything that has been written here but a lot of it also makes a lot of sense. I would love to read a little more about it.
I had got a desire to begin my own organization, however I did not earn enough of cash to do this.
These facts are amazing . I was searching before last 5 weaks and ia din’t get the perfect answer. But after all i found from your site. thanks for posting such a interesting topic.منتديات بنات
You did a good job explaning the code. I like how it is well commented or documented as some say. Keep at it.
hy you are doing a great job. i was looking for this information. i found it on your page its really amazing. I like that information. I am sure that these are your own views. They are really awsome. You are really ffantastic. thanks
Your article is written very content, All of the projects look great! Keep the Ball Rolling. Thanks!
It is not true suffering ennobles the character; happiness does that sometimes, but suffering, for the most part, makes men petty and vindictive. Let us suggest to the person in crisis that he cease concentrating so upon the dangers involved and the difficultie，and concentrate instead upon the opptunity—-for there is always opportunity in crisis.
Happy to see your blog as it is just what I’ve looking for and excited to read all the posts. I am looking forward to another great article from you.
It is not true suffering ennobles the character; happiness does that sometimes, but suffering, for the most part, makes men petty and vindictive. download the hangover 2 | download kung fu panda 2
interesting article, thanks for sharing :)
greffe de cheveux
A really nice article.good story and it’s very interesting and useful article.It is not true suffering ennobles the charter.Gladness dose not some time but suffering is the most important part.So i like that your work.keep it up.[url=http://justinbieber.info/]Justin Bieber[/url].
What rails? I do the html screen scraping..
Happy to see your blog, and that is just what I looking for and excited to read all messages. I’m looking forward to another great article from you.Happy to see your blog, and that is just what I looking for and excited to read all messages. I’m looking forward to another great article from you.
I really enjoyed reading this post. Keep up the good work.
Thanks for this post, good job!
Thanks for giving us objective information, much appreciate this!