Data, data everywhere

I have recently read an interesting article from the Economist entitled “Data, data everywhere”. The author has regrouped some interesting (and impressive) figures regarding amount of data. I learned that astronomy is certainly the domain where most data are generated:

WHEN the Sloan Digital Sky Survey started work in 2000, its telescope in New Mexico collected more data in its first few weeks than had been amassed in the entire history of astronomy.

An important conclusion is that it will soon be needed to preprocess data before storing them. This is a necessity since the increase in data is bigger than the increase in storage capacity. One of the issue, according to me, is how to select or aggregate useful data before using them. Since you don’t always know in advance how you will use the data, it may be very difficult to perform this preprocessing to save disk space. Any thought on this topic?

Read the full article from the Economist.

Share

Recommended Reading

Comments Icon38 comments found on “Data, data everywhere

  1. Hey,

    I agree, that the problem of information overload has rise significantly.
    And the major problem comes in data preprocessing as the data is available in different sources and different forms (structured, semi-structured and unstructured)
    Recently i got to know about an Informatica ETL tool. I think this tool could be useful in preprocessing of structured data from may different sources.
    It lets you know the different properties of data automatically.
    For example: for each column it tells us the percentage of unique values, null values, its inferred data type, simple patterns present for each column, statistics (max length, min length, top 5 values, bottom 5 values) and many such information.
    It also lets you automatically store data in databases in structured format.

    I think it could be a very helpful tool for data-preprocessing.

  2. Very interesting blog. Alot of blogs I see these days don’t really provide anything that I’m interested in, but I’m most definately interested in this one. Just thought that I would post and let you know.

  3. Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you.

  4. Nice to be visiting your blog again, it has been months for me. Well this article that i’ve been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.

  5. Great info! I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have

  6. In your blog I was happy to see your article, better than last time, and have made great progress, I am very pleased. I am looking forward to your article will become better and better.

  7. I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I am always searching like this type blog post. I hope I will see again..

  8. I found lots of interesting information here.Great work
    Thanks for the share loved reading the article, please do share more like this wiht us .

  9. I’m selfish, impatient and a little insecure. I make mistakes, I am out of control and at times hard to handle. But if you can’t handle me at my worst, then you sure as hell don’t deserve me at my best

  10. To all the true lovers: Your unconditional love is the reason why flowers bloom. To all the beloveds: You are the reason why the universe came into existence

  11. Great article and subject! Thanks 😀 It’s an amazing post. Thank you again! It sounds like good. You are gathering lots of different ideas in your blog. Good work my friends. Instead of thinking about what you haven’t written, look at how much you HAVE written 🙂

Comments are closed.