Fetch data from web sites
Why do we collect statistics?
Essentially, we collect statistics to gain information that helps us make the right decisions. Sometimes we collect statistics to visualize and learn more about the specific domain (classical motivation), sometimes for future analysis (business intelligence (BI) motivation). Both of these motivations helps gain a business edge - know something that others don't. Having a business edge will help you in your research/work, which have the potential to earn you more money and ultimately to secure your future.
How do we begin to secure our future?
Pick something to do. Be good at what you do. Collect statistics for studies. Improve by learning.
Can we collect statistics off Internet?
The Internet is one place to harvest data. For 20 years ago, in 1990, the Internet was a desert. Universities in some countries used it. Since then, more and more people, organizations and businesses have put more and more data onto Internet in one form or another. Some are freely accessible, some requires subscription. The data presented can be raw data or it can be ready statistical elements, like indicators, averages or spread. More than we might expect can be counted and included into statistics; list sizes, single numbers found in a listing or in detail pages. Since many sites present their information in a machinery, straightforward manner, it is easy to scan and harvest by another machinery.
Okay, I'm ready! How do we go about hunting down the meat?
Why would I tell you for free? I'm earning my living by helping you with exactly this kind of stuff! However, since I'm a really positive, mind-sharing kind of guy, I will give you elements of such a procedure and hope that someone, someday will need me as part of their project.
In this case, go straight to my Data Collecting Tools Review (under construction).
Essentially, we collect statistics to gain information that helps us make the right decisions. Sometimes we collect statistics to visualize and learn more about the specific domain (classical motivation), sometimes for future analysis (business intelligence (BI) motivation). Both of these motivations helps gain a business edge - know something that others don't. Having a business edge will help you in your research/work, which have the potential to earn you more money and ultimately to secure your future.
How do we begin to secure our future?
Pick something to do. Be good at what you do. Collect statistics for studies. Improve by learning.
Can we collect statistics off Internet?
The Internet is one place to harvest data. For 20 years ago, in 1990, the Internet was a desert. Universities in some countries used it. Since then, more and more people, organizations and businesses have put more and more data onto Internet in one form or another. Some are freely accessible, some requires subscription. The data presented can be raw data or it can be ready statistical elements, like indicators, averages or spread. More than we might expect can be counted and included into statistics; list sizes, single numbers found in a listing or in detail pages. Since many sites present their information in a machinery, straightforward manner, it is easy to scan and harvest by another machinery.
Okay, I'm ready! How do we go about hunting down the meat?
Why would I tell you for free? I'm earning my living by helping you with exactly this kind of stuff! However, since I'm a really positive, mind-sharing kind of guy, I will give you elements of such a procedure and hope that someone, someday will need me as part of their project.
- Analyze. What do you want to find? (objective)
- Analyze. Where do you expect to find it? (sources)
- Analyze. Why do you want to it? (ultimate goal)
- Analyze. Read retrospectives summaries of similar, earlier projects
- Analyze. Validate correct direction (no alternative feasible routes to ultimate goal)
- Quality. Report peer review. Retrospective
- Design. Evaluate existing solutions (low cost, bad match)
- Design. Initial custom solution sketch (high cost, custom suit)
- Design. Revised custom solution sketch (medium cost, mix reuse/custom-built, good match)
- Design. Purpose-built graphical design sketch (custom design)
- Design. Detailed project plan until first milestone
- Quality. Report peer review. Retrospective
- Implement. Development
- Quality. Unit test
- Quality. Integration test
- Quality. Beta-test
- Quality. Education
- Quality. Acceptance test
- Implement. Production
- Quality. Project Retrospective
- Maintenance
- Quality. Revision. Retrospective
In this case, go straight to my Data Collecting Tools Review (under construction).