http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html?_r=0
Yet far too much handcrafted work — what data scientists call “data wrangling,” “data munging” and “data janitor work” — is still required. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.
ClearStory Data, a start-up in Palo Alto, Calif., makes software that recognizes many data sources, pulls them together and presents the results visually as charts, graphics or data-filled maps. Its goal is to reach a wider market of business users beyond data masters.
Trifacta makes a tool for data professionals. Its software employs machine-learning technology to find, present and suggest types of data that might be useful for a data scientist to see and explore, depending on the task at hand.
Showing posts with label data mining. Show all posts
Showing posts with label data mining. Show all posts
Monday, August 18, 2014
Tuesday, November 5, 2013
Slides and Video: Metrics that Matter with Chris Geoghegan
http://www.net2van.com/2013/09/slides-and-video-metrics-that-matter-with-chris-geoghegan/
Chris has bad news. Most of the time you spend looking at your analytics is probably in vain. There is too much data that tells you too little information. You thought analytics was all about using information to make better decisions, but it isn’t working. How do you go beyond just giving yourself a nice pat on the back with how many visitors you got last month?
Chris has bad news. Most of the time you spend looking at your analytics is probably in vain. There is too much data that tells you too little information. You thought analytics was all about using information to make better decisions, but it isn’t working. How do you go beyond just giving yourself a nice pat on the back with how many visitors you got last month?
Thursday, December 13, 2012
Hard-Wired for Chocolate and Hybrid Cars? How Genetics Affect Consumer Choice
http://www.sciencedaily.com/releases/2010/09/100920173004.htm
The authors studied twins' consumer preferences to determine whether or not certain behaviors or traits have a genetic basis. "A greater similarity in behavior or trait between identical than between fraternal twins indicates that the behavior or trait is likely to be heritable," the authors explain.
"The current research suggests that heritable and other hard-wired inherent preference components play a key role in behavior and deserve much more attention in marketing and decision-making research," the authors write.
Itamar Simonson and Aner Sela. On the Heritability of Consumer Decision Making: An Exploratory Approach for Studying Genetic Effects on Judgment and Choice.Journal of Consumer Research, April 2011 [link]
http://www.theglobeandmail.com/news/national/time-to-lead/why-your-dna-is-a-goldmine-for-marketers/article6293064/
As for Prof. Pike, he believes marketers should have an individual’s explicit consent to use their DNA to customize ads. “If the choice is between random ads … and targeted ads that might actually have some appeal, then I’d probably prefer to be exposed to the latter.” He’s skeptical that marketers will ever be able to use his genetic profile to discern his taste in vacations, but he’s certain of one thing: “Marketers are expert at data mining, and if there is data that can be mined then I’m confident that they’ll try to do so.”
The authors studied twins' consumer preferences to determine whether or not certain behaviors or traits have a genetic basis. "A greater similarity in behavior or trait between identical than between fraternal twins indicates that the behavior or trait is likely to be heritable," the authors explain.
"The current research suggests that heritable and other hard-wired inherent preference components play a key role in behavior and deserve much more attention in marketing and decision-making research," the authors write.
Itamar Simonson and Aner Sela. On the Heritability of Consumer Decision Making: An Exploratory Approach for Studying Genetic Effects on Judgment and Choice.Journal of Consumer Research, April 2011 [link]
http://www.theglobeandmail.com/news/national/time-to-lead/why-your-dna-is-a-goldmine-for-marketers/article6293064/
As for Prof. Pike, he believes marketers should have an individual’s explicit consent to use their DNA to customize ads. “If the choice is between random ads … and targeted ads that might actually have some appeal, then I’d probably prefer to be exposed to the latter.” He’s skeptical that marketers will ever be able to use his genetic profile to discern his taste in vacations, but he’s certain of one thing: “Marketers are expert at data mining, and if there is data that can be mined then I’m confident that they’ll try to do so.”
Tuesday, December 4, 2012
Kaggle - INFORMS Data Mining Contest
http://www.kaggle.com/c/informs2010
http://www.kaggle.com/
We’re making data science a sport.™
Kaggle is an arena where you can match your data science skills against a global cadre of experts in statistics, mathematics, and machine learning. Whether you're a world-class algorithm wizard competing for prize money or a novice looking to learn from the best, here's your chance to jump in and geek out, for fame, fortune, or fun.
http://www.kaggle.com/
We’re making data science a sport.™
Kaggle is an arena where you can match your data science skills against a global cadre of experts in statistics, mathematics, and machine learning. Whether you're a world-class algorithm wizard competing for prize money or a novice looking to learn from the best, here's your chance to jump in and geek out, for fame, fortune, or fun.
Tuesday, April 17, 2012
Children of the 90s: Coming of age
http://www.nature.com/news/children-of-the-90s-coming-of-age-1.10396?WT.ec_id=NEWS-20120417
“It's the deepest phenotyping and biobank resource of any large birth cohort.”
Avon Longitudinal Study of Parents And Children (ALSPAC)
http://www.bristol.ac.uk/alspac/
http://en.wikipedia.org/wiki/Avon_Longitudinal_Study_of_Parents_and_Children
This library is the harvest from an unusual study of humanity. In 1990, researchers started to collect tissues and detailed information from more than 14,500 pregnant women in this western British city and its surrounding region of Avon. The women filled in more than 100 pages of questionnaires about their health, relationships, work and home. After birth, researchers tracked the children's development through surveys, clinical examinations and biological samples. They know what the kids ate, when they first talked, how often they fell sick and when a parent read to them — or deserted them. They know when the children started to hit puberty, drink alcohol, have sex and leave home. In that wealth of data — collected at a cost of some £42 million (US$67 million) so far — they are tracing how genetic and environmental factors in the children's early years affect their later ones.
One showed that eating oily fish during pregnancy was associated with better eye and cognitive development in children5, 6. Another helped to cement advice that babies should be put to sleep on their backs to reduce the risk of cot death, by showing that this sleeping position did not cause any developmental delays7. A third showed the first association between peanut allergy — an emerging epidemic in Western countries — and peanut oil in baby lotions8. Manufacturers soon started identifying the ingredient on labels.
“It's the deepest phenotyping and biobank resource of any large birth cohort.”
Avon Longitudinal Study of Parents And Children (ALSPAC)
http://www.bristol.ac.uk/alspac/
http://en.wikipedia.org/wiki/Avon_Longitudinal_Study_of_Parents_and_Children
This library is the harvest from an unusual study of humanity. In 1990, researchers started to collect tissues and detailed information from more than 14,500 pregnant women in this western British city and its surrounding region of Avon. The women filled in more than 100 pages of questionnaires about their health, relationships, work and home. After birth, researchers tracked the children's development through surveys, clinical examinations and biological samples. They know what the kids ate, when they first talked, how often they fell sick and when a parent read to them — or deserted them. They know when the children started to hit puberty, drink alcohol, have sex and leave home. In that wealth of data — collected at a cost of some £42 million (US$67 million) so far — they are tracing how genetic and environmental factors in the children's early years affect their later ones.
One showed that eating oily fish during pregnancy was associated with better eye and cognitive development in children5, 6. Another helped to cement advice that babies should be put to sleep on their backs to reduce the risk of cot death, by showing that this sleeping position did not cause any developmental delays7. A third showed the first association between peanut allergy — an emerging epidemic in Western countries — and peanut oil in baby lotions8. Manufacturers soon started identifying the ingredient on labels.
Thursday, August 25, 2011
Using biomedical networks to prioritize gene–disease associations
http://www.dovepress.com/articles.php?article_id=8144
Understanding the genetic foundations of genetic diseases, such as cancer, Alzheimer disease, or Huntington’s disease, is critical to the development of new diagnostics and treatments. Several computational methods have been used to speed up the discovery process, eg, by selecting the molecular targets for a given disease. However, despite the achievements obtained over recent years, better solutions are still required. This paper presents an innovative computational method that addresses the problem of using disperse biomedical knowledge to select the best candidate genes associated with a disease. The method uses a network representation of current biomedical knowledge that includes biomolecular concepts such as genes, diseases, pathways, and biological process. It also applies information extraction techniques to enrich the network with more dynamic and updated data. A biologically inspired algorithm is applied to this network in order to identify association levels between genes and diseases. The solution proposed here surpasses many limitations of previous methods such as the need for training data. The validation applied demonstrates that the proposed method has best overall results compared with state-of-the-art methods as it also performs especially well for the critical top-rank positions. We believe this method represents a major advance over previous work and that it will be a key tool for future gene–disease association studies.
gene–disease, biomedical networks, prioritization, computational method
Understanding the genetic foundations of genetic diseases, such as cancer, Alzheimer disease, or Huntington’s disease, is critical to the development of new diagnostics and treatments. Several computational methods have been used to speed up the discovery process, eg, by selecting the molecular targets for a given disease. However, despite the achievements obtained over recent years, better solutions are still required. This paper presents an innovative computational method that addresses the problem of using disperse biomedical knowledge to select the best candidate genes associated with a disease. The method uses a network representation of current biomedical knowledge that includes biomolecular concepts such as genes, diseases, pathways, and biological process. It also applies information extraction techniques to enrich the network with more dynamic and updated data. A biologically inspired algorithm is applied to this network in order to identify association levels between genes and diseases. The solution proposed here surpasses many limitations of previous methods such as the need for training data. The validation applied demonstrates that the proposed method has best overall results compared with state-of-the-art methods as it also performs especially well for the critical top-rank positions. We believe this method represents a major advance over previous work and that it will be a key tool for future gene–disease association studies.
gene–disease, biomedical networks, prioritization, computational method
Monday, August 8, 2011
Kendall tau distance rank correlation coefficient
Specifically, it is a measure of rank correlation: that is, the similarity of the orderings of the data when ranked by each of the quantities. It is named after Maurice Kendall, who developed it in 1938,[1] though Gustav Fechner had proposed a similar measure in the context of time series in 1897.[2]
http://en.wikipedia.org/wiki/Kendall_tau_rank_correlation_coefficient
Rank aggregation methods -- Shili Lin
http://onlinelibrary.wiley.com/doi/10.1002/wics.111/pdf
Kendall’s Tau top-k distance for ranking set of genes, classification or clustering of datasets based on patterns of differential expression
theory.stanford.edu/~sergei/papers/www10-metrics.pdf
For top k lists T1
and T2, the minimizing Kendall distance Kmin(T1,T2)
between rl and v2 is defined to be the minimum value
of K ( a l , a 2 ) , where al and a2 are each permutations of
DT1 U DT2 and where al >= T1 and a2 >= T2
http://en.wikipedia.org/wiki/Kendall_tau_rank_correlation_coefficient
Rank aggregation methods -- Shili Lin
http://onlinelibrary.wiley.com/doi/10.1002/wics.111/pdf
Kendall’s Tau top-k distance for ranking set of genes, classification or clustering of datasets based on patterns of differential expression
theory.stanford.edu/~sergei/papers/www10-metrics.pdf
For top k lists T1
and T2, the minimizing Kendall distance Kmin(T1,T2)
between rl and v2 is defined to be the minimum value
of K ( a l , a 2 ) , where al and a2 are each permutations of
DT1 U DT2 and where al >= T1 and a2 >= T2
Tuesday, August 2, 2011
Inferring cellular networks using probabilistic graphical models.
http://www.ncbi.nlm.nih.gov/pubmed/14764868
1. Science. 2004 Feb 6;303(5659):799-805.
Inferring cellular networks using probabilistic graphical models.
Friedman N.
School of Computer Science and Engineering, Hebrew University, 91904 Jerusalem,
Israel. nir@cs.huji.ac.il
High-throughput genome-wide molecular assays, which probe cellular networks from
different perspectives, have become central to molecular biology. Probabilistic
graphical models are useful for extracting meaningful biological insights from
the resulting data sets. These models provide a concise representation of complex
cellular networks by composing simpler submodels. Procedures based on
well-understood principles for inferring such models from data facilitate a
model-based methodology for analysis and discovery. This methodology and its
capabilities are illustrated by several recent applications to gene expression
data.
PMID: 14764868 [PubMed - indexed for MEDLINE]
1. Science. 2004 Feb 6;303(5659):799-805.
Inferring cellular networks using probabilistic graphical models.
Friedman N.
School of Computer Science and Engineering, Hebrew University, 91904 Jerusalem,
Israel. nir@cs.huji.ac.il
High-throughput genome-wide molecular assays, which probe cellular networks from
different perspectives, have become central to molecular biology. Probabilistic
graphical models are useful for extracting meaningful biological insights from
the resulting data sets. These models provide a concise representation of complex
cellular networks by composing simpler submodels. Procedures based on
well-understood principles for inferring such models from data facilitate a
model-based methodology for analysis and discovery. This methodology and its
capabilities are illustrated by several recent applications to gene expression
data.
PMID: 14764868 [PubMed - indexed for MEDLINE]
Boosting Signal-to-Noise in Complex Biology: Prior Knowledge Is Power
Trey Ideker, Janusz Dutkowski, and Leroy Hood, “Boosting Signal-to-Noise in Complex Biology: Prior Knowledge Is Power,” Cell 144, no. 6 (March 2011): 860-863.
http://www.cell.com/abstract/S0092-8674(11)00244-3
- filter
- integrators: combine many weak signals to increase statistical power
Examples
- Roach et al., 2010 - Familial SNV study found 3 candidate genes for Miller syndrome
- protein signaling network of kinases and phosphotases: McGary et al., 2010 - same network across species control diff. phenotypes and diff. network show similar phenotype (Erwin and Davidson 2009)
http://www.cell.com/abstract/S0092-8674(11)00244-3
- filter
- integrators: combine many weak signals to increase statistical power
Examples
- Roach et al., 2010 - Familial SNV study found 3 candidate genes for Miller syndrome
- protein signaling network of kinases and phosphotases: McGary et al., 2010 - same network across species control diff. phenotypes and diff. network show similar phenotype (Erwin and Davidson 2009)
Monday, April 18, 2011
Condtional Random Field - CRF (Machine Learning)
http://en.wikipedia.org/wiki/Conditional_random_field
A conditional random field (CRF) is a type of discriminative undirected probabilistic graphical model. It is most often used for labeling or parsing of sequential data, such as natural language text or biological sequences[1] and computer vision[2] . Specifically, CRFs find applications in shallow parsing[3] , named entity recognition[4] and gene finding, among other tasks, being an alternative to the related hidden Markov models.
http://www.inference.phy.cam.ac.uk/hmw26/crf/
The primary advantage of CRFs over hidden Markov
models is their conditional nature, resulting in the relaxation of the indepen-
dence assumptions required by HMMs in order to ensure tractable inference.
CRFs outperform both MEMMs
and HMMs on a number of real-world sequence labeling tasks
Nando de Freitas
http://www.cs.ubc.ca/~nando/
http://www-stat.stanford.edu/~tibs/ElemStatLearn/download.html
David JC MacKay
http://www.inference.phy.cam.ac.uk/mackay/itila/
A conditional random field (CRF) is a type of discriminative undirected probabilistic graphical model. It is most often used for labeling or parsing of sequential data, such as natural language text or biological sequences[1] and computer vision[2] . Specifically, CRFs find applications in shallow parsing[3] , named entity recognition[4] and gene finding, among other tasks, being an alternative to the related hidden Markov models.
http://www.inference.phy.cam.ac.uk/hmw26/crf/
The primary advantage of CRFs over hidden Markov
models is their conditional nature, resulting in the relaxation of the indepen-
dence assumptions required by HMMs in order to ensure tractable inference.
CRFs outperform both MEMMs
and HMMs on a number of real-world sequence labeling tasks
Nando de Freitas
http://www.cs.ubc.ca/~nando/
http://www-stat.stanford.edu/~tibs/ElemStatLearn/download.html
- Deep Learning Website.
- Geoff Hinton's Website.
- Yann Lecun's Website.
- Andrew Ng's Website.
- Jason Weston's Website.
- Russ Salakhutdinov's Website.
- The book of Kevin Murphy.
- Hastie, Tibshirani and Friedman: The elements of statistical learning.
- Machine learning video lectures
- Why stats: NYTimes article
- The following handout should help you with linear algebra revision: PDF
David JC MacKay
http://www.inference.phy.cam.ac.uk/mackay/itila/
Tuesday, November 2, 2010
Analytic Bridge
Data mining, statistics, quant, operations research, six sigma, econometrics, web analytics, text mining, business intelligence, SAS, biostatistics, machine learning, artificial intelligence, decision sciences, cloud computing, SaaS.
http://www.analyticbridge.com/
Subscribe to:
Posts (Atom)