Just a collection of some random cool stuff. PS. Almost 99% of the contents here are not mine and I don't take credit for them, I reference and copy part of the interesting sections.
Tuesday, March 30, 2021
PySpark Demo
Download SSL / TLS certificates using Python
import ssl
hostname='www.google.com'
port=443
cert = ssl.get_server_certificate((hostname, port))
x509 = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
x509.digest('sha256')
# b'0A:E6:46:01:80:9F:C7:33:84:19:6A:DD:6C:8E:5F:95:5C:F6:F2:75:46:32:1E:C9:61:1D:88:DA:9A:A9:B4:
x509.get_subject()
# <X509Name object '/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com'
Monday, March 29, 2021
Kafka
Kafka is an asynchronous messaging queue. Kafka consumer, consumes message from Kafka and does some processing like updating the database or making a network call.
https://kafka.apache.org/documentation/What is Apache Kafka? Why is it so popular? Should you use it?
https://techbeacon.com/app-dev-testing/what-apache-kafka-why-it-so-popular-should-you-use-it
What, why and How Apache Kafka
https://www.startdataengineering.com/post/what-why-and-how-apache-kafka/
Kafka partitions
https://www.educba.com/kafka-partition/
Getting started with Apache Kafka in Python
https://towardsdatascience.com/getting-started-with-apache-kafka-in-python-604b3250aa05
How can Kafka consumers parallelise beyond the number of partitions
PySpark Streaming
Connecting the Dots (Python, Spark, and Kafka)
https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/
Parquet format at Twitter
Compressed columnar format (per Row group < 1GB) for Hadoop
Efficient scan with compression. Parquet supports deeply nested structures, efficient encoding and column compression schemes, and is designed to be compatible with a variety of higher-level type systems.
https://www.youtube.com/watch?v=Qfp6Uv1UrA0
Hybrid storage model (horizontal row groups and vertical column chunks partitioning)
https://www.youtube.com/watch?v=1j8SdS7s_NY
Row group filtering through predicate pushdown. Each row group has min-max stats for each predicate for fast filtering. Use dictionary filtering to filter by exact values.
Optimization: avoid many small files (too many stats/footer, overhead) and few huge files (>1 GB?). Need to repartition data.
Automatic repartition using Delta Lake.
Spark Reading and Writing to Parquet Storage Format
https://www.youtube.com/watch?v=-ra0pGUw7fo
df.select('mycol','bla').write.partitionBy('mycol').mode(SaveMode.Append).format('parquet').save('/tmp/foo')
How Adobe Does 2 Million Records per second using Apache Spark
https://www.youtube.com/watch?v=rPgoPHAEYAM
val kafka = spark.readStream.format('kafka').option(...).load()
Read Parquet Files in Python
https://www.youtube.com/watch?v=XFO5jdGsMek
pip install pandas
pip install pyarrow
pd.read_parquet(parquet_file, engine='auto')
*parquet_file can be a single parquet file or a folder of parquet files
Thursday, March 25, 2021
Boyer-Moor Exact Pattern Matching Algorithm
https://www.youtube.com/watch?v=4Xyhb72LCX4
Boyer–Moore string-search algorithm - Wikipedia
O(n + m) where n and m are lengths of p (search pattern) and t (text to search)
advantages
- skip alignments
- doesn't compare all the characters
Try alignments from left-to-right and compare characters in the alignment from right-to-left
Bad character rule - if a mismatched character is found going from right-to-left then shift the pattern to the right until a matching character in the pattern is found
Good suffix rule - shift until the good matching suffix matches
Try both rules and shift by the largest amount of shift
number of characters to shift is stored in a lookup table which is created using only the search pattern
The algorithm preprocesses the string being searched for (the pattern), but not the string being searched in (the text). It is thus well-suited for applications in which the pattern is much shorter than the text or where it persists across multiple searches. The Boyer–Moore algorithm uses information gathered during the preprocess step to skip sections of the text, resulting in a lower constant factor than many other string search algorithms. In general, the algorithm runs faster as the pattern length increases. The key features of the algorithm are to match on the tail of the pattern rather than the head, and to skip along the text in jumps of multiple characters rather than searching every single character in the text.
Wednesday, March 24, 2021
Randomly sample files
From https://unix.stackexchange.com/questions/108581/how-to-randomly-sample-a-subset-of-a-file
perl -ne 'print if (rand() < .01)' huge_file.csv > sample.csv
Tuesday, March 23, 2021
BOTORCH: A Framework for Efficient Monte-Carlo Bayesian Optimization
BOTORCH: A Framework for Efficient Monte-Carlo Bayesian Optimization
To address these problems, we developed BoTorch, a framework for Bayesian optimization research, and Ax, a robust platform for adaptive experimentation. BoTorch follows the same modular design philosophy as PyTorch, which makes it very easy for users to swap out or rearrange individual components in order to customize all aspects of their algorithm, thereby empowering researchers to do state-of-the art research on modern Bayesian optimization methods. By exploiting modern parallel computing paradigms on both CPUs and GPUs, it is also fast.
Core Data Science researchers discuss research award opportunity in adaptive experimentation
Sunday, March 21, 2021
SQL Challenge
https://www.hackerrank.com/challenges/revising-the-select-query/problem
Easy
1. Query all columns for all American cities in the CITY table with populations larger than 100000. The CountryCode for America is USA.
select * from City where population > 100000 and CountryCode='USA'
2. Query the NAME field for all American cities in the CITY table with populations larger than 120000. The CountryCode for America is USA.
The CITY table is described as follows:
select Name from City where population > 120000 and CountryCode='USA'
3. Query a list of CITY names from STATION for cities that have an even ID number. Print the results in any order, but exclude duplicates from the answer.
select distinct City from Station where mod(id,2)=0
Medium
1. Query an alphabetically ordered list of all names in OCCUPATIONS, immediately followed by the first letter of each profession as a parenthetical (i.e.: enclosed in parentheses). For example: AnActorName(A), ADoctorName(D), AProfessorName(P), and ASingerName(S).
select * from (select concat('There are a total of ',count(occupation),' ',lower(occupation),'s.') as stat from occupations group by occupation) as innertable order by innertable.stat;
2. Pivot the Occupation column in OCCUPATIONS so that each Name is sorted alphabetically and displayed underneath its corresponding Occupation. The output column headers should be Doctor, Professor, Singer, and Actor, respectively.
Note: Print NULL when there are no more names corresponding to an occupation.
https://codingsight.com/pivot-tables-in-mysql/
User nobuh:
select
max(if(occupation='Doctor', name, null)) as docname,
max(case when occupation='Professor' then name else null end) as profname,
max(case when occupation='Singer' then name else null end) as singername,
max(case when occupation='Actor' then name else null end) as actorname
from (
select
a.occupation,
a.name,
(select count(name) from occupations b where a.occupation=b.occupation and a.name > b.name) as name_rank
from occupations a
) as rank_summary
group by name_rank
order by name_rank
name_rank:
selecta.occupation,a.name,-- get the name rank by comparing the names(select count(name) from occupations b where a.occupation=b.occupation and a.name > b.name) as name_rankfrom occupations aorder by name_rank;
Doctor Aamina 0Actor Eve 0Singer Christeen 0Professor Ashley 0Doctor Julia 1Singer Jane 1Professor Belvet 1Actor Jennifer 1Actor Ketty 2Singer Jenny 2Doctor Priya 2Professor Britney 2Singer Kristeen 3Professor Maria 3Actor Samantha 3Professor Meera 4Professor Naomi 5Professor Priyanka 6
3. You are given a table, BST, containing two columns: N and P, where N represents the value of a node in Binary Tree, and P is the parent of N.
Write a query to find the node type of Binary Tree ordered by the value of the node. Output one of the following for each node:
Root: If node is root node.
Leaf: If node is leaf node.
Inner: If node is neither root nor leaf node.
Answer
select * from bst;select distinct '------' from bst;select * from (select n,'Root' from bst bo where bo.p is nullunion allselect n,'Inner' from bst bo where bo.p is not null and bo.n in (select bi.p from bst bi where p is not null)union allselect n,'Leaf' from bst bo where bo.p is not null and bo.n not in (select bi.p from bst bi where p is not null)) tmporder by n;
1 23 25 67 62 46 44 158 910 912 1314 139 1113 1111 1515 NULL------1 Leaf2 Inner3 Leaf4 Inner5 Leaf6 Inner7 Leaf8 Leaf9 Inner10 Leaf11 Inner12 Leaf13 Inner14 Leaf15 Root
4. Given the table schemas below, write a query to print the company_code, founder name, total number of lead managers, total number of senior managers, total number of managers, and total number of employees. Order your output by ascending company_code.
https://www.hackerrank.com/challenges/the-company/problem
SQL
select c.company_code,founder
,count(distinct lead_manager_code)
,count(distinct senior_manager_code)
,count(distinct manager_code)
,count(distinct employee_code)
from company c left join employee e on c.company_code=e.company_code group by c.company_code,founder
order by cast(substr(c.company_code,2,5) as UNSIGNED);
Output
C1 Monika 1 2 1 2
C2 Samantha 1 1 2 2
https://www.hackerrank.com/challenges/weather-observation-station-20/problem
A median is defined as a number separating the higher half of a data set from the lower half. Query the median of the Northern Latitudes (LAT_N) from STATION and round your answer to decimal places.
*/
select format(lat_n,4) from (select s1.id,s1.lat_n,(select count(*) from station),(select cast(count(id)/2 as unsigned) from station) as mid_rank,(select count(s2.id) from station s2 where s1.lat_n > s2.lat_n) as latn_rank from station s1 order by s1.lat_n) as rank_tablewhere (latn_rank +1 )= mid_rank;
select round(s.lat_n,4) from station s where (select round(count(s.id)/2)-1 from station) = (select count(s1.id) from station s1 where s1.lat_n > s.lat_n);
83.8913
6. https://www.hackerrank.com/challenges/interviews/problem
https://www.w3schools.com/sql/func_mysql_date_add.asp
SELECT DATE_ADD("2017-06-15", INTERVAL 10 DAY); # returns 2017-06-25
Saturday, March 20, 2021
Wednesday, January 10, 2018
Having persistent / intermittent / sporadic Wifi disconnections with Windows 10?
How to disable Wifi sense



Wednesday, April 29, 2015
Thursday, March 12, 2015
Points of Significance in Biology - Nature column
http://www.nature.com/collections/qghhqm
