Migrating code from Zeppelin to Spark

When you have shiny Zeppelin application, which runs smoothly and does what it supposed to do, you start transferring your code into Spark environment to use it in production. If you are novice in Hadoop environment (like me), you might encounter a couple of tasks, required to be solved before you will celebrate project launch.

Basically, it can be broken down into easy chunks:

  1. Launching spark-submit with test class.
  2. Adding main class and Spark context initialization.
  3. Building fat jar (which includes all the  libraries).
  4. Launching a job with a spark-submit.



Founders at work — short summary

One of the best books about the history of the common internet, how the whole IT sphere was developed, through the interviews of the founders of tech companies. It is unique material of opinions and real life stories of founders companies which made a lot of today commodities.

This is a book, there can be no good short summary for, because every story is unique, but they have something in common. And despite that it can only be highly recommended to read this book, despite the fact that most of the companies were built and sold over 2 decades ago, there are some key insights which can be written down here.

  1. Doing something new and innovative isn’t fun. Usually it takes much more than 40 hours work week. And there is only one way to allocate more time — from sleep.
  2. Business plans are useless. Life is too complicated and the only purpose of writing a business is demonstrating that you are committed enough to do some extra work.
  3. People is everything that matters. Team and connections — this is one of the fundamental things, which if crucial to performing the best.
  4. Users are never speaking of their problems, they are suggesting feature they think can help them.

Замена дисплея на маке по гарантии от Staingate в России

Staingate — это признанная Apple проблема, из-за которой экраны (а точнее — антибликовое покрытие) MacBook-ов начинали выглядеть потёртыми или поцарапанными, не смотря на отсутствие явных физических повреждений. Проблемы были подвержены ноутбуки от 2013 до 2015 выпуска и для них существует возможность бесплатной замены дисплея в авторизованных центрах Apple. Не смотря на отсутствие фирменных магазинов, в России работают авторизованные сервисные центры, в которые можно придти и совершенно бесплатно заменить дисплей.



Nagios doesn’t start / internal service error / PID unfound

Nagios web-interface can fail to connect to process or to display any content at all. There can be different source of the problem, but the key to solving it:

  1. Checking permissions on the cgi-bin, fcgi-bin folders and scripts inside /nagios path.
  2. Checking Suexec log if you use it (`/var/log/apache2/suexec.log` for example).
  3. Checking access mode on statusjson.cgi and status.cgi

In case it doesn’t help, and nagios simply refuses to start, showing only:

Running configuration check...
Starting nagios:.

You have to manually run configuration check with the -v flag:

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If the check is correct, but nags refuses to run, providing no info in {path}/var/nagios.log and not creating {path}/var/nagios.lock, then the problem might be in the device space.


Setting up Nagios monitoring on Debian-based servers

Nagios is a great tool for server monitoring, providing rich environment to setup different notifications and monitoring scenarios.

It supports hosts and notification service grouping, comes with a GUI out-of-the-box and has really great community and lots of documentation online.

Pretty much all the setup can be done, based on the google’s top search results. Nice place to start is Digital Ocean post about setting up Nagios on Ubuntu 14.04. There is  also another nice post, explaining how configuration is done.

If you fail to install package, or your server is different from Ubuntu, f.e, you run some Debian Stable server, you can address this post (RU) and build Nagios from source.

Configuring client can be done via this post, but check_nrpe command available only on server, not on the client.


Short summary of «So Good They Can’t Ignore You»

This is a book about finding your mission. In this book not only author introduced a framework for dealing with mission-follow-your-passion-work-has-to-bring-pleasure problem, but he also wrote how he applied the advices, he came up with during the research, for himself.



Short summary of «Talent Is Overrated»

Geoff Colvin’s «Talent Is Overrated» is a great complimentary book to the famous «The Flow» and represents the results of the study of process of achieving outstanding results in different areas. TLDR: there is no talent, only grit.



Running cv2 in virtualenv on Mac

You may install cv2 using one of the available guides, f.e. this one.

And then just symlink so files into required virtualenv:

ln -s /usr/local/opt/opencv/lib/python2.7/site-packages/cv2.so /Users/user/.virtualenv/virtualenv/lib/python2.7/site-packages/


Why reading matters

Reading matters. Invention of writing let people preserve and pass knowledge and things that matter through generation. So reading was basically designed to save people from making mistakes someone had already made, to save people from going down the paths, which are already explored. But the obvious reason — to gain knowledge and experience which otherwise would be more expensive to gain — is not the only why lots of media advice so much how you can read one book a week.

Reading can help you to form habit of completion, which is extremely important, because conscientiousness, associated with completion by definition,  is the best predictor of many important human outcomes, including mortality, longevity, educational attainment, and a host of criteria related to career success, or at least so stated in «Organized mind».



Notes about bayesian probability and statistical measures

Statistical measures of performance can be really confusing sometimes, especially when explained in terms of sickness. So I came up with a Google Spreadsheet which shows example of standard statistical measures of performance and bayesian probabilities.