Kafka

Kafka is an asynchronous messaging queue. Kafka consumer, consumes message from Kafka and does some processing like updating the database or making a network call.

https://kafka.apache.org/documentation/

What is Apache Kafka? Why is it so popular? Should you use it?

https://techbeacon.com/app-dev-testing/what-apache-kafka-why-it-so-popular-should-you-use-it

What, why and How Apache Kafka

https://www.startdataengineering.com/post/what-why-and-how-apache-kafka/

Kafka partitions

https://www.educba.com/kafka-partition/

Getting started with Apache Kafka in Python

https://towardsdatascience.com/getting-started-with-apache-kafka-in-python-604b3250aa05

How can Kafka consumers parallelise beyond the number of partitions

https://medium.com/@jhansireddy007/how-can-kafka-consumers-parallelise-beyond-the-number-of-partitions-a0a46ade8a6c

PySpark Streaming

https://stackoverflow.com/questions/62342080/how-to-programmatically-load-and-stream-kafka-topic-to-a-pyspark-dataframe#62342332

Connecting the Dots (Python, Spark, and Kafka)

https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/

Parquet format at Twitter

Compressed columnar format (per Row group < 1GB) for Hadoop

Efficient scan with compression. Parquet supports deeply nested structures, efficient encoding and column compression schemes, and is designed to be compatible with a variety of higher-level type systems.

https://www.youtube.com/watch?v=Qfp6Uv1UrA0

Hybrid storage model (horizontal row groups and vertical column chunks partitioning)

https://www.youtube.com/watch?v=1j8SdS7s_NY

Row group filtering through predicate pushdown. Each row group has min-max stats for each predicate for fast filtering. Use dictionary filtering to filter by exact values.

Optimization: avoid many small files (too many stats/footer, overhead) and few huge files (>1 GB?). Need to repartition data.   

Automatic repartition using Delta Lake.

Spark Reading and Writing to Parquet Storage Format

https://www.youtube.com/watch?v=-ra0pGUw7fo

df.select('mycol','bla').write.partitionBy('mycol').mode(SaveMode.Append).format('parquet').save('/tmp/foo')

How Adobe Does 2 Million Records per second using Apache Spark

https://www.youtube.com/watch?v=rPgoPHAEYAM

val kafka = spark.readStream.format('kafka').option(...).load()

Read Parquet Files in Python

https://www.youtube.com/watch?v=XFO5jdGsMek

pip install pandas

pip install pyarrow

pd.read_parquet(parquet_file, engine='auto')

*parquet_file can be a single parquet file or a folder of parquet files

For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights

http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html?_r=0
 
Yet far too much handcrafted work — what data scientists call “data wrangling,” “data munging” and “data janitor work” — is still required. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

ClearStory Data, a start-up in Palo Alto, Calif., makes software that recognizes many data sources, pulls them together and presents the results visually as charts, graphics or data-filled maps. Its goal is to reach a wider market of business users beyond data masters.

Trifacta makes a tool for data professionals. Its software employs machine-learning technology to find, present and suggest types of data that might be useful for a data scientist to see and explore, depending on the task at hand.

On the scalability of statistical procedures: why the p-value bashers just don't get it.

http://simplystatistics.org/2014/02/14/on-the-scalability-of-statistical-procedures-why-the-p-value-bashers-just-dont-get-it/

I think of particular interest given the NIH Director's recent comments on reproducibility is our course on Reproducible Research. There are also many more specialized resources that are very good and widely available that will build on the base we created with the data science specialization.

1. For scientific software engineering/reproducibility: Software Carpentry.
2. For data analysis in genomics: Rafa's Data Analysis for Genomics Class.
3. For Python and computing: The Fundamentals of Computing Specialization

LabKey Software

http://medcitynews.com/2014/01/open-sourced-seattle-fated-collision-software-talent-life-science-researchers/#ixzz2rLZYJd7B

LabKey Server is open source software that helps scientists organize, analyze, and share biomedical research data.

Read more: http://medcitynews.com/2014/01/open-sourced-seattle-fated-collision-software-talent-life-science-researchers/#ixzz2rLvIbCgM

https://labkey.org/

Taking R to the Limit (High Performance Computing in R)

http://www.slideshare.net/bytemining/r-hpc

bigmemory - it is ideal for problems involving the analysis in R for manageable subsets of the data, or when an analysis is conducted mostly in C++

the "big" family

biganalytics, synchronicity, bigtabulate, big.matrix, bigalgebra, bigvideo, shared.big.matrix, filebacked.big.matrix, bigsplit

linear models: biglm.big.matrix

mwhich

ff - "fast access files" - file-based access to datasets that cannot fit in memory

data.table

mapReduce - apply(map(data), reduce)

HadoopStreaming

http://www.ncbi.nlm.nih.gov/pubmed/21210976?dopt=Abstract

BMC Bioinformatics. 2010 Dec 21;11 Suppl 12:S1. doi: 10.1186/1471-2105-11-S12-S1.

An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics.

Taylor RC.

Author information

Abstract

BACKGROUND:

Bioinformatics researchers are now confronted with analysis of ultra large-scale data sets, a problem that will only increase at an alarming rate in coming years. Recent developments in open source software, that is, the Hadoop project and associated software, provide a foundation for scaling to petabyte scale data warehouses on Linux clusters, providing fault-tolerant parallelized analysis on such data using a programming style named MapReduce.

DESCRIPTION:

An overview is given of the current usage within the bioinformatics community of Hadoop, a top-level Apache Software Foundation project, and of associated open source software projects. The concepts behind Hadoop and the associated HBase project are defined, and current bioinformatics software that employ Hadoop is described. The focus is on next-generation sequencing, as the leading application area to date.

CONCLUSIONS:

Hadoop and the MapReduce programming paradigm already have a substantial base in the bioinformatics community, especially in the field of next-generation sequencing analysis, and such use is increasing. This is due to the cost-effectiveness of Hadoop-based analysis on commodity Linux clusters, and in the cloud via data upload to cloud vendors who have implemented Hadoop/HBase; and due to the effectiveness and ease-of-use of the MapReduce method in parallelization of many data analysis algorithms.

PMID:

 

21210976

 

[PubMed - indexed for MEDLINE] 

PMCID:

 

PMC3040523

 

Free PMC Article

GridGain - In-memory Big Data

Process TBs of data on 1000s of nodes with in-memory speed anddatabase reliability using Java or Scala.

http://www.gridgain.com/
http://www.gridgain.com/blog/gridgain-hadoop-differences-synergies/

GridGain is Java-based middleware for in-memory processing of big data in a distributed environment. It is based on high performance in-memory data platform that integrates fast In-Memory MapReduce implementation with In-Memory Data Grid technology delivering easy to use and easy to scale software. Using GridGain you can process terabytes of data, on 1000s of nodes in under a second.

GridGain typically resides between business, analytics, transactional or BI applications and long term data storage such as RDBMS, ERP or Hadoop HDFS, and provides in-memory data platform for high performance, low latency data storage and processing.

Both, GridGain and Hadoop, are designed for parallel processing of distributed data. However, both products serve very different goals and in most cases are very complementary to each other. Hadoop is mostly geared towards batch-oriented offline processing of historical and analytics payloads where latencies and transactions don’t really matter, while GridGain is meant for real-time in-memory processing of both transactional and non-transactional live data with very low latencies.

 

Big data

http://www.nature.com/nature/journal/v493/n7433/full/493473a.html?WT.ec_id=NATURE-20130124

To get the best out of big data, funding agencies should develop shared tools for optimizing discovery and train a new breed of researchers, says Chris A. Mattmann.