serhii.net

Quotes

Get things out of your head and into a system that you fully trust. Everything you do should have positive value – it’s either improving you (I put self care and genuine leisure time in here, but not time wasting), improving a relationship, making money, or making one of those other things more efficient. Do high energy and high focus things when you actually have energy and focus; do mindless things when you feel mindless. Do not skimp on self-care, which includes genuine leisure time, good healthy food, exercise, good personal relationships, and adequate sleep. Aim for the “flow state” in everything you do, because you’ll never be better than when you’re so engaged that you lose track of time and place and just get lost in the moment. (How I get things done)

I find that forcing myself to think about those things at the pace of my handwriting brings a ton of clarity to the ideas I’m struggling with or the life issues I’m trying to figure out. (same source)

it’s easy to sleep well when you get up early and work hard. (same source)

“No more yes. It’s either HELL YEAH! or no.” — Derek Sivers

Random

I need a system to consistently track things I’m trying to optimize in my life. Today I already read N articles about excellent things I can do with my life, and usually it would end at it. Probably the first in line would be reinforcement and mental contrasting.

On a certain level we actually bump aganst the infinitely familiar thing about not knowing what I want.

The plan

• From now on, if I read something motivational in the morning, it should be one thing. And focus on it, think on it, only on it.

DNB and Typing

460 cpm 98%

d4b 14% Thu 18 Apr 2019 12:54:55 PM CEST
d4b 0% Thu 18 Apr 2019 12:56:50 PM CEST
d4b 11% Thu 18 Apr 2019 12:58:46 PM CEST
d3b 85% Thu 18 Apr 2019 01:00:22 PM CEST !
d4b 50% Thu 18 Apr 2019 01:03:42 PM CEST
d4b 17% Thu 18 Apr 2019 01:05:37 PM CEST
d4b 50% Thu 18 Apr 2019 01:07:32 PM CEST
d4b 61% Thu 18 Apr 2019 01:09:28 PM CEST
d4b 67% Thu 18 Apr 2019 01:11:25 PM CEST
d4b 50% Thu 18 Apr 2019 01:13:19 PM CEST

Pandas

I’m familiar with most of this, but since I find myself googling it every time, I’ll just write it here, so I’ll know where to loo.

Scipy-lectures.org

Scipy Lecture Notes seems like a very interesting place.

Concatenate dafaframes

pd.concat([d, dd]) concatenates them leaving the same columns. pd.concat([d, dd], ignore_index=True) concatenates them leaving the same columns and having a common id column. pd.concat([d, dd], axis=1) merges them horizontally, that is there will be all the columns from the input dataframes.

Seaborn multiple distplots on the same graph

The article

Seaborn plt and labeling

Apparently sns.plt is a bug which has been fixed. Nice. Regardless, the new correct way is import matplotlib.pyplot as plt; plt.....

Pandas multiple conditions filtering

dsa[ (dsa.char_count>190) & (dsa.char_count<220) ]

Jupyter – making cells 100% wide

from IPython.core.display import display, HTML display(HTML("<style>.container { width:100% !important; }</style>")) inside a cell (SO)

Thesis

I have my semi-final dataset, today I’ll clean it, analyze, and output it to some clean.csv file. Along with creating a script that cleans the data, for all the repetitive things I’ll have to do.

Analyzing the dataset

0418-analysis-of-final-dataset.

What I did

• Added quite a lot of features.
  • token_count != pos_count.
  • Counts of POS are relative.
• Currently I have many more UK tweets than others - but I should have at least 10000 tweets for each language.

Interesting stuff

• Twitter does not count @replies in its character count
  • This is why sometimes we get such bundles of joy of 964 characters:

{%raw%}’@FragrantFrog @BourgeoisViews @SimonHowell7 @Mr_Bo_Jangles_1 @Joysetruth @Caesar2207 @NancyParks8 @thetruthnessie @carmarsutra @Esjabe1 @DavidHuddo @rob22_re @lindale70139487 @anotherviv @AndyFish19 @Jules1602xx @EricaCantona7 @grand___wazoo @PollyGraph69 @CruftMs @ZaneZeleti @McCannFacts @ditsy_chick @Andreamariapre2 @barragirl49 @MancunianMEDlC @rambojambo9 @MrDelorean2 @Nadalena @LoverandomIeigh @cattywhites2 @Millsyj73 @strackers74 @may_shazzy @JBLittlemore @Tassie666 @justjulescolson @regretkay @Chinado59513358 @Louise42368296 @TypRussell @Anvil161Anvil16 @DuskatChristie @McCannCaseTweet @noseybugger1 @HilaryDean15 @DesireeLWiggin1 @M47Jakeman @crocodi11276514 @jonj85014 If it was in the Scenic several weeks after she was reported missing.Her body must have been put there.!\nWho by ?The people who hired the Scenic ! How hard is that to understand ?\nThis algorithmic software gives a probability of the identity of each contributer to the sample !\n😏’{%endraw%}

• Otherwise, we get a pretty similar distribution. Except also the 200 characters effect that’s especially pronounced in SA - do they use old clients or something similar?

Now playing: The Godfather II Soundtrack

Possible ideas for additional cleanup

• I can just remove from the text the all the @mentions except the first two. That would still give me a difference between replying to one or to multiple people, but I would assume would fare much better with various NLI stuffs.

Stack

Add search to this blog via this simple js

To watch: Hacking democracy with theater

Quotes

It was a small Army Security Agency Station in Southeast Asia that I was doing some work for. They had a shrink and he pulled me aside. In just 10 minutes or so he taught me “breathing”. It wasn’t until the internet that I learned the term mindful breathing. Subsequently I figured out it was some sort of meditation. [..]\ \ He said I was ‘wrapped to tight’. What ever that means. Those guys were all spooks, but I did not have the same clearances. I was an outsider in that regard, but I did eat with them when at their place. I guess he was bored.\ \ He took my blood pressure and then taught me to breathe. Then he took it again. I was surprised at the drop. It hooked me on mindful breathing. It was probably a parlor trick, but it worked. He improved my lifetime health. For that I thank him.\ (from reddit)

Linux PDF forms

Okular can fill and save PDF forms. Zathura can open already filled forms.

Converting PDF to PNG, much better method than convert

pdftoppm input.pdf outputname -png\ pdftoppm input.pdf outputname -png -f {page} -singlefile It works much better than convert.

Timewarrior

timeww continue continues the last tracked thing

Python multiline comments

Even though stylistically questionable (PEP8 favours multiple multiline comments), one possibility is to use """ mycomment """; when they are not a docstring they are ignored. (source). They have to be indented right though. And feel kinda wrong\ Additionally:

triple-quotes are a way to insert text that doesn’t do anything (I believe you could do this with regular single-quoted strings too), but they aren’t comments - the interpreter does actually execute the line (but the line doesn’t do anything). That’s why the indentation of a triple-quoted ‘comment’ is important. – Demis Jun 9 ‘15 at 18:35

The Internet is wonderful

• Daily links from the recurse.com community
• Archivebox is a way to archive webpages you care about in a future-proof way. I really should get all the links from my Link wiki via a script and just download everything. Yay for Data hoarding!

Thesis

This is an excellent paper about Reddit and more focused on orthoographic errors. Will read next! \ And this is an awesome annotated dataset, exactly the kind I need.

SSH

Exiting a dead SSH session

SSH can handle commands. From the blog post above: <Enter>~.\ SSH parses commands sent after a newline and ~. ~. is the one to exit.

SSH config files

In ~/.ssh/config.

Host host1
    HostName ssh.example.com
    User myuser
    IdentityFile ~/.ssh/id_rsa

allows to just do sh host1.

… Still amazed by Linux and the number of such things. If I ever planned to do Linux much more professionally, I would just sit and read through all the man pages of the typical tools, systematically.

Stack

I need to make this Diensttagebuch searchable from the website, not just locally with :Ag.

Stack

Taskwarrior negating filters

t id!=123, works with everything.

Python2

For unicode strings, do “unicode string”.encode(‘utf-8’)

Thesis

Current results

I looked again at the confusion matrix, after having made a copy. It’s quite interesting:

array([[29, 14, 28, 26],
       [38, 57, 36, 27],
       [52, 18, 58, 28],
       [18, 14, 18, 39]])

This is a simple SVM, using extremely simple features, and 2000 examples per class. The columns/rows are: ar, jp, lib, it, in that order. My first error is that Arabic and countries which are around Libya are quite similar in my world, linguistically, and we can see that they are confused quite often, in both directions. Italy and Japan do much better.

• Get more and better (linguistically more different) data.
• Work with more interesting features.

Still, ich finde das sehr vielversprechend, and definitely better than chance. And logically it makes sense. I’ll continue.

Countries with the most Twitter users

The list. I’ll stick to Japan, UK, SA, Brazil, India – quite between each other, geographically and linguistically. I leave the US alone, too mixed.

Bounding boxes

This is the picker. DublinCore format is in the identical order as Twitter wants!

Probably the plan would be

• Getting the dataset
  • Except the 5 languages I already have, add a similar one to the ones already available, to see how much confusion between the two I get at the end.
    • Added Mexico!
• Preprocessing
  • Replace URLs and @mentions by tags.
  • Replace the actual words with their POS Tags
    • Leaving the Emoticons alone, since they are probably quite geographically distant
    • Leaving the usual punctuation and stop-words alone, since they probably are exactly what I need
  • Remove all usernames which contain ‘bot’ in their username
  • Find all tweets that are similar to each other by whatever metric and remove all of them too
    • This would work much better than what I could manually do, I can’t think of all possible robotic tweets
  • Then tokenize the resulting thing the usual way
• Ensemble learning
  • I can get a number of classifiers and use some kind of voting procedure
  • BoW is counterproductive in my case, because too many geographical names and topic names. BUT it would be fascinating to get tweets from the same authors a number of years before, and compare if BoW gets less effective for these old tweets. I think it would be too focused for the ephemeral Twitter universe, if there’s an election in Brazil it will happily decide that all tweets containing ’election’ are Brazilian - a comparison with old tweets would help me test this hypothesis. And give the user a choice at the end if the prediction should be done using everything or everything except BoW.

To research

• Author profiling
  • By what markers is this usually done? Can I use some of them?

For tomorrow/later

• Finish doing the preprocessing script
  • In: the .csv
  • Out: Whatever I can import in Jupyter, with all the features etc

Pandas

Leave rows with values from a certain list

d[d.co.isin(['uk','in'])] leaves the rows where co==‘uk’ or co==‘in’. \ For multiple conditions, df.loc[(df['column_name'] >= A) & (df['column_name'] <= B)]\ TODO: Why is .loc used here?

Random/thoughts

• Would putting an uninterrupted block of learning at the very beginning of my day help me?
• This might become a very nice experiment – do it for 30 days and see what happens. If I sleep well I’m on my best in the mornings, apparently.
• Publishing papers with markdown

ssh

Has a config file! This opened a new universe for me too.

Nearlyfreespeech ssh via public key

The key needs to be added from the panel, adding it to the user folder as usual does not work.

German

Wann vs wenn

Wann vs wenn: Wann has nothing to do with if, it’s a question asking for a point of time. Wenn is closer to “if”, but it’s also a translation for “when”.

If we can say at what point time instead of when, then we need to use wann.

Wann [=at what time/when] kommt der Bus? \ Bis wann musst du arbeiten? \ Thomas fragt Maria, wann genau sie nach Hause kommt.

On the other hand, \ Ich gehe nach Hause wenn[!= at what time! just the “when” closer to “if”] ich fertig bin.

Roles

A wann-clause is ALWAYS functioning as the object of the verb.. If I can replace the clause with a thing, then it’s wann.\ Wenn answers to “at what time”, we can basically replace it with “at 3 am”.

When I have finished work, I will call you and tell you when I will be at home.\ When I have finished work, I will call you and tell you at what point in time I will be at home.\ Wenn ich mit der Arbeit fertig bin, rufe ich dich an und sage dir, wann ich zuhause bin.\ At 3 I’ll call you and tell you this thing.

Github reset (undoing last commit/s)

$ git reset --soft HEAD~1 resets to last commit leaving all the changes on disc, but uncommitted. \ $ git reset --hard 0ad5a7a6 returns to any previous version.

.gitignore for LaTeX projects

Here, and it’s excellent. I should actually learn git in a normal systematic way. Additionally, what to do when your .gitignore is ignored by git@SO.

Random

Busy person patterns as linked on HN Testosterone seems to have different effects than the stereotypes say, and road/roid rage is actually caused by estrogen spikes.

Cooking

This eggs inside avocado recipe is very interesting. Will try tomorrow. Also this avocado hummus recipe.

DNB and Typing

d4b 33% Sun 07 Apr 2019 04:24:36 PM CEST
d4b 33% Sun 07 Apr 2019 04:26:35 PM CEST
d4b 56% Sun 07 Apr 2019 04:28:28 PM CEST
d4b 61% Sun 07 Apr 2019 04:30:24 PM CEST
d4b 28% Sun 07 Apr 2019 04:32:21 PM CEST
d4b 44% Sun 07 Apr 2019 04:34:27 PM CEST
d4b 22% Sun 07 Apr 2019 04:36:19 PM CEST
d4b 39% Sun 07 Apr 2019 04:38:14 PM CEST

Quotes

“Wherever you are, make sure you’re there.” — Dan Sullivan

Diploma

Classifying by parts of speech

nltk.download() downloads everything needed. nltk.word_tokenize('aoethnsu') returns the tokens. From [https://medium.com/@gianpaul.r/tokenization-and-parts-of-speech-pos-tagging-in-pythons-nltk-library-2d30f70af13b](This article). For parts of speech it’s nltk.pos_tag(tokens).

The tokenizer for twitter works better for URLs (of course). Interestingly it sees URLs as NN. And - this is actually fascinating - smileys get tokenized differently!

 ('morning', 'NN'),
 ('✋', 'NN'),
 ('🏻', 'NNP'),

EDIT: nltk.tokenize.casual might be just like the above, but better!

EDIT: I have a column with the POS of the tweets! How do I classify it with its varying length? How can I use the particular emojis as another feature?

Ideas

POS + individual smileys might be enough for it to generalize! TODO test TODO: Maybe first do some much more basic feature engineering with capitalization and other features mentioned here:

    Word Count of the documents – total number of words in the documents
    Character Count of the documents – total number of characters in the documents
    Average Word Density of the documents – average length of the words used in the documents
    Puncutation Count in the Complete Essay – total number of punctuation marks in the documents
    Upper Case Count in the Complete Essay – total number of upper count words in the documents
    Title Word Count in the Complete Essay – total number of proper case (title) words in the documents
    Frequency distribution of Part of Speech Tags:
        Noun Count
        Verb Count
        Adjective Count
        Adverb Count
        Pronoun Count

Resources

textminingonline.com has nice resources on topic which would be very interesting to skim through! Additionally flair is a very interesting library not to reinvent the wheel, even though reinventing the wheel would be the entire point of a bachelor’s thesis.

This could work as a general high-levent intro into NLP? Also this.

Stack

Edit .i3/ to create the multiple scratchpads at startup and put them automatically where I want them – second answer is a good example.

DNB and Typing

450 cpm 97%

d4b 72% Fri 05 Apr 2019 07:03:22 PM CEST
d4b 50% Fri 05 Apr 2019 07:05:21 PM CEST
d4b 39% Fri 05 Apr 2019 07:07:23 PM CEST
d4b 44% Fri 05 Apr 2019 07:09:19 PM CEST
d4b 33% Fri 05 Apr 2019 07:11:17 PM CEST
d3b 79% Fri 05 Apr 2019 07:13:08 PM CEST !
d3b 71% Fri 05 Apr 2019 07:14:44 PM CEST !
d3b 86% Fri 05 Apr 2019 07:16:21 PM CEST !
d4b 44% Fri 05 Apr 2019 07:18:17 PM CEST
d4b 22% Fri 05 Apr 2019 07:20:13 PM CEST
d4b 28% Fri 05 Apr 2019 07:22:41 PM CEST
d4b 00% Fri 05 Apr 2019 07:24:46 PM CEST

Random

Knowledge/plans management

I just discovered didoesdigital.com, which is absolutely excellent on all levels. I’m missing a way to categorize everything I see there.

I should/could make things-I’m-learning pages with links and checklist for things I’m doing/learning. I’m not quite sure what should it look like, but it would definitely be something Jekyll-like. I think I’m slowly going in the direction of Steve Wolfram’s dashboard. Or at least a different vim in a different floating window that opens with another keystroke, i3 would make it easy to do that. In general I need a much better system to track the things I’m learning or reading. Polarized goes in the right direction. And I feel my links wiki will stay just that – a links wiki. Unless I make a seamless interface to it, I don’t really like it for actual knowledge management, even though it’s the absolute best I have until now.

And I must not fall in my typical error about sharpening the saw more that actually cutting trees, even though sharpening the saw is a really pleasant thing to do for me.

EDIT: Just created it at here, we’ll see what happens. I can imagine a dashboard based on it, and some kind of integration for task/timewarrior. Probably something ncurses-based in python?

Trickle lists

This is the application - in general I find the idea really inspiring. I could imagine it on a touchscreen somewhere, or at least on a second desktop. Is it conceptually different from Nomie? Can I add just add another “trickle” board?

Jekyll deploy.sh

Added at the end ./commit.sh, which is a small file with git commit, so now it gets backed up to github automatically every time I deploy a new version on the server.

Stack

DNB and Typing

d4b 44% Sun 31 Mar 2019 11:42:18 AM CEST
d4b 50% Sun 31 Mar 2019 11:44:21 AM CEST
d4b 17% Sun 31 Mar 2019 11:46:18 AM CEST
d4b 6% Sun 31 Mar 2019 11:48:20 AM CEST
d4b 39% Sun 31 Mar 2019 11:50:20 AM CEST
d4b 17% Sun 31 Mar 2019 11:52:47 AM CEST
d4b 17% Sun 31 Mar 2019 11:54:49 AM CEST
d4b 67% Sun 31 Mar 2019 11:56:52 AM CEST
d4b 56% Sun 31 Mar 2019 11:59:03 AM CEST
d4b 39% Sun 31 Mar 2019 12:01:05 PM CEST
d4b 6% Sun 31 Mar 2019 12:03:29 PM CEST
d4b 44% Sun 31 Mar 2019 12:05:30 PM CEST

d4b 39% Sun 31 Mar 2019 02:52:21 PM CEST
d4b 50% Sun 31 Mar 2019 02:54:35 PM CEST
d4b 44% Sun 31 Mar 2019 02:56:44 PM CEST
d4b 44% Sun 31 Mar 2019 02:58:43 PM CEST
d4b 44% Sun 31 Mar 2019 03:00:46 PM CEST
d4b 39% Sun 31 Mar 2019 03:03:16 PM CEST
d4b 44% Sun 31 Mar 2019 03:05:19 PM CEST
d4b 39% Sun 31 Mar 2019 03:07:16 PM CEST

Taskwarrior

Tasks tagged +next are now underlined.