How to Extract Data from an Alternated Text Like the Internet?

Published 16 November 06 10:01 PM | sirpadk

Many ppl believe that using regex to extract data from a web page is a suicide task. the text changes rapidly, you'll never be able to know what will change, and how, you are risking saving the wrong information is some of the arguments these ppl will tell you.

Well let me get one thing strait, I'm working in a company which is making money from using regex to extract information from hundreds of websites. those websites do change, but because we are thinking very good, and analyzing the site before we jump and write a regex, we can make this impossible mission, very easy.

As I said before, the first rule is Analyzing Analyzing A-n-a-l-y-z-i-n-g! Look on the source code, and look for anchors, and I don't mean <a> tags, but look for text, or tags that you know will less likely to change. a class attribute in an HTML tag will change, a message or a banner will change, but if the site I'm extracting data from is a CD shop site, and there is a lable saying Price:, than this will probably won't change, and if it does, than we can say that the site had a major change, and we will probably need to write more than one regex from scratch.

The second rule is "Better no data than wrong data" write you regex not too flexible, as you wouldn't want to save the ISBN number of a book as it's price.

The 3rd rule is zooming. it performs better, and it minimize risks for mistaking data. Zooming is a process where we start with isolating an area of information with a little regex that is using an anchor "Price:(.*?)</table>" this regex will give us a smaller area of text, which we know for sure has the information we need, and not all the other just that may confuse our regex. now we can write a detailed regex, or even better many small regex, that will allocate the information we need. the performance will be very good as the text is very small, and the risk of extracting the zip code of the company address as a price is 0.

Well that's all for tonight, there are many other rules that i follow, and I'll share them later on.

May the force be with you

sirpadk

Sponsor

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

# Jack said on November 18, 2006 12:34 PM:

Hey, a few people I know reckon this proves reg exp wrong: tags like img or a href or tags like img or a href so I make this to try prove them wrong: p = Pattern.compile((.*?)() matcher.group(2) - doesn't work! :( sorry, but I disagree with you any suggestion to prove them wrong would be appreciated PS if you make it pattern.compile() pattern.compile(matcher.group(0) + (.*?)()) then you have so many lines of code that you may as well use substring and indexOf methods instead

# Serge said on November 30, 2006 1:51 AM:

> I'm working in a company which is making money from using regex to extract information from hundreds of websites. I would like to work in such company too. :-)

# erytre said on May 23, 2008 2:27 AM:

The fourth <a href="http://www.wow-powerleveling.org">wow power leveling</a> latest game in <a href="http://www.xowow.com">wow power leveling</a> Warcraft series is ‘<a href="http://www.powerlevelingweb.com">wow power leveling</a>’. Also known as <a href="http://www.igsstar.com">wow power leveling</a>, it represents a <a href="http://www.wowgoldweb.com">wow power leveling</a> multiplayer online <a href="http://www.powerleveling-wow.com/siteMap.asp">wow power leveling</a> game, the best of <a href="http://www.powerleveling-wow.com">wow power leveling</a> kind. Initially, it was <a href="http://www.wow-powerleveling.org">wow gold</a> it be released in 2001, but <a href="http://www.powerleveling-wow.com">wow powerleveling</a> was delayed <a href="http://www.powerlevelingweb.com">wow powerleveling</a> 2004, thus <a href="http://www.xowow.com">wow powerleveling</a> the 10 years of<a href="http://www.wow-powerleveling.org">wow powerleveling</a> franchise of this<a href="http://www.xowow.com">wow gold</a> series. The <a href="http://www.powerlevelingweb.com">world of warcraft power leveling</a> was not <a href="http://www.wow-powerleveling.org/wow+power+leveling.html">world of warcraft power leveling</a>fulfilling, because <a href="http://www.powerleveling-wow.com/siteMap.asp">wow power level</a>problems with <a href="http://www.powerlevelingweb.com">wow power level</a> server’s stability <a href="http://www.powerleveling-wow.com/siteMap.asp">power leveling wow</a> performance occurred, but <a href="http://www.xowow.com">power leveling wow</a> game still <a href="http://www.powerlevelingweb.com">power leveling wow</a> a financial success <a href="http://www.powerleveling-wow.com/siteMap.asp">powerleveling wow</a> the most <a href="http://www.xowow.com">powerleveling wow</a> game of its kind. The number <a href="http://www.powerlevelingweb.com">cheap wow power leveling </a> users that play <a href="http://www.igsstar.com/Maple-Story-mesos-us/">Maple Story mesos</a>, exceeds 8.5 <a href="http://www.igsstar.com/Maple-Story-mesos-us/">MapleStory mesos</a>, worldwide.As a form <a href="http://www.igsstar.com/Maple-Story-mesos-us/">ms mesos</a>,recognition for <a href="http://www.igsstar.com/Maple-Story-mesos-us/">mesos</a>,outstanding popularity, the game <a href="http://www.igsstar.com/Silkroad-Online-Gold/">SilkRoad Gold</a>, received a<a href="http://www.igsstar.com/Silkroad-Online-Gold/">SRO Gold</a>, of awards. Now the question <a href="http://www.igsstar.com/EverQuest-2-gold-plat/">eq2 plat</a>, why is <a href="http://www.igsstar.com/EverQuest-2-gold-plat/">eq2 gold</a>, game <a href="http://www.igsstar.com/EverQuest-2-gold-plat/">eq2 Platinum</a>, popular? For anyone<a href="http://www.igsstar.com/EverQuest-2-gold-plat/">EverQuest 2 Platinum</a>, played the previous <a href="http://www.igsstar.com/EverQuest-2-gold-plat/">EverQuest 2 gold</a>, and <a href="http://www.igsstar.com/EverQuest-2-gold-plat/">EverQuest 2 plat</a>, already initiated <a href="http://www.igsstar.com/Lord-of-The-Ring-online-gold/">lotro">http://www.igsstar.com/Lord-of-The-Ring-online-gold/">lotro gold</a>, the mysterious world <a href="http://www.igsstar.com/Lord-of-The-Ring-online-gold/">lotr gold</a>, the breathtaking <a href="http://www.igsstar.com/Lord-of-The-Ring-online-gold/">Lord of the Rings online Gold</a>, this <a href="http://www.watchreplicashop.com/">Rolex Replica</a> nothing but an <a href="http://www.watchreplicashop.com/">Replica Rolex</a> adventure that continues the story of ‘Warcraft III: Frozen Throne’, four years after conclusion, in the world of Azeroth. The game is online role-playing, the previous versions being online and offline strategy games. The major thrills and unique features are present as in every Blizzard game.

# tactical led said on June 19, 2008 4:55 AM:

<a href="http://www.flashlightcool.com/">Tactical Flashlights</a>

<a href="http://www.flashlightcool.com/">Tactical Flashlight</a>

<a href="http://www.flashlightcool.com/">Flashlight</a>

<a href="http://www.flashlightcool.com/">Led Flashlight</a>

<a href="http://www.flashlightcool.com/">Led Flashlights</a>

<a href="http://www.flashlightcool.com/">Led Lights</a>

<a href="http://www.flashlightcool.com/">Rechargeable Flashlights</a>

<a href="http://www.flashlightcool.com/">Streamlight</a>

<a href="http://www.flashlightcool.com/">Tactical Gear</a>

<a href="http://www.flashlightcool.com/">Tactical Light</a>

<a href="http://www.flashlightcool.com/">Tactical Lights</a>

<a href="http://www.flashlightcool.com/">Xenon Flashlight</a>

<a href="http://www.flashlightcool.com/">Xenon Flashlights</a>

<a href="http://www.flashlightcool.com/">Palights</a>

<a href="http://www.flashlightcool.com/">HID Flashlight</a>

<a href="http://www.flashlightcool.com/">High-intensity Flashlights</a>

<a href="http://www.flashlightcool.com/">Wolf-Eyes Flashlight</a>

<a href="http://www.flashlightcool.com/">Wolf-Eyes Flashlights</a>

<a href="http://www.rcroll.com/">rc">http://www.rcroll.com/">rc helicopter</a>

<a href="http://www.rcroll.com/">helicopter</a>

<a href="http://www.rcroll.com/">airplane</a>">http://www.rcroll.com/">airplane</a>

<a href="http://www.rcroll.com/">r/c</a>">http://www.rcroll.com/">r/c</a>

<a href="http://www.rcroll.com/">rc">http://www.rcroll.com/">rc</a>

<a href="http://www.rcroll.com/">rc">http://www.rcroll.com/">rc airplane</a>

<a href="http://www.rcroll.com/">rc">http://www.rcroll.com/">rc heli</a>

<a href="http://www.rcroll.com/">r c helicopter</a>

<a href="http://www.rcroll.com/">r c airplane</a>

<a href="http://www.rcroll.com/">airplane model</a>

<a href="http://www.rcroll.com/">remote control</a>

<a href="http://www.game4box.com/">video game</a>

<a href="http://www.game4box.com/">Playstation 2</a>

<a href="http://www.game4box.com/">Playstation 3</a>

<a href="http://www.game4box.com/">Nintendo DS</a>

<a href="http://www.game4box.com/">Gamecube</a>

# JAck said on June 27, 2008 10:44 PM:

http://www.blackjack21onlinestrategy.com/how_to_play_blackjack.html

Thanks for the backlink, hope it works!

# cc said on July 18, 2008 11:09 PM:

http://www.warheroonline.com

# bwrewerew said on July 25, 2008 2:08 AM:

[url=http://www.vipaocgold.com/] Beer aoc gold[/b][/url][url=http://www.aocsale.com/] Beer aoc gold[/b][/url][url=http://www.aocsale.com/] Beer age of conan gold[/b][/url][url=http://www.buyfastgold.com/] Beer aoc gold[/b][/url][url=http://www.aocgold10.com/] Beer age of conan gold[/b][/url][url=http://buy-aoc-gold.hellgate-pd.com/] Beer aoc gold[/b][/url][url=http://buy-aoc-gold.rgtrcredit.com/] Beer age of conan gold[/b][/url]

<A href="http://www.vipaocgold.com/" >aoc gold</A>  <A href="http://www.aocsale.com/" >aoc gold</A>  <A href="http://www.aocsale.com/" >age of conan gold</A>   <A href="http://www.buyfastgold.com/" >aoc gold</A>  <A href="http://www.aocgold10.com/">age of conan gold</A>  <A href="http://buy-aoc-gold.hellgate-pd.com/">aoc gold</A> <A href="http://buy-aoc-gold.rgtrcredit.com/">age of conan gold</A>

# lvbugai said on August 26, 2008 1:23 AM:

<A href="http://www.storeingame.com">wow gold</A> and <A href="http://www.storeingame.com/aoc-gold.htm"> <A href="http://www.storeingame.com">wow gold</A> and <A href="http://www.storeingame.com/aoc-gold.htm">aoc gold</A>,our website is <A href="http://www.storeingame.com/wow-gold.htm">wow gold</A> Professional offered,please buy <A href="http://www.2joygame.com">lotro gold</A> in www.storeingame.com.

<A href="http://www.wzyizhou.com/jizhuangdai.htm">集装袋</A>aoc">http://www.wzyizhou.com/jizhuangdai.htm">集装袋</A>aoc gold</A>,our website is <A href="http://www.storeingame.com/wow-gold.htm">wow gold</A> Professional offered,please buy <A href="http://www.2joygame.com">lotro gold</A> in www.storeingame.com.

Name (required)
Your URL (optional)
Comments (required)
Remember Me?
Enter the code you see below

Regex Jedi

How to Extract Data from an Alternated Text Like the Internet?

Comment Notification

Comments

Leave a Comment

Search

This Blog

Tags

Navigation

Archives

Syndication