serhii.net

Transformer & BERT ML article

NLP — BERT & Transformer - Jonathan Hui - Medium is probably the best explanation of the Transformer and BERT I’ve even seen.

Fill paragraph, Knuth and psychologically bad breaks

Implement Fill Paragraph for coments · Issue #2737 · intellij-rust/intellij-rust

This is fascinating problem, originally solved by Knuth himself, and a primarily example of dynamic programming: www.eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf. However interesting the problem might be, I’d rather my IDE solved it, and not me :-)

TODO - read it sometime maybe, I’ve skimmed through it and it looks very interesting.

Intellij Idea ideavim plugin remove keybinding

Predictably to remove a keybinding imported from .vimrc in .ideavimrc the usual syntax is used: unmap <C-h>

Intellij Idea Ideavim go to last modified line

The vim shortcut '' works in ideavim as well!

General idea about Tensorflow and writing stuff for it

It’d be nice to have fixed values (and especially dimensions!) for my tensors that I use for testing stuff such as layers. If my input tensors’ dimensions are predictable, like [10, 9, 8, ...] for each dimension, I’d be able to recognize them immediately at various stages without having to think about it. For a level up, I can consider for example Fibonacci numbers or similar, that have different divisors, so that the program will fail early and predictably if I do something wrong.

Otherwise I kinda like the approach of Tensor Considered Harmful, especially the part where dimensions have human-readable names.

Deutsch / Jour fixe

TIL: Jour fixe ist ein in einer kleinen Gruppe von Personen fest vereinbarter, regelmäßig wiederkehrender Termin (Regeltermin), zum Beispiel „jeder zweite Donnerstag im Monat“. In der Arbeitswelt wird zu diesem Regeltermin meist eine Besprechung durchgeführt.

reStructuredText for Python documentation

7. Documenting Python — Python Developer’s Guide is something I can try to follow. PEP 287 – reStructuredText Docstring Format | Python.org has more examples.

Github permanent link to file

Getting permanent links to files - GitHub Help tells me to press y for the URI to update to the permanent link to that version of the file.

Intellij idea ideavim search

Ideavim search seems “smart” by default, ignores case unless I use case in my search.

Ambient sleeping pill

Ambient Sleeping Pill – Internet Radio for Sleep, Meditation & Relaxation is what it says it is. Background music is okay. “For sleep, meditation or study”. Nice alternative to brain.fm

Intellij idea “Add as content source” getting ignored that leads to “module not found” in Python depending on the order of the files of $PYTHONPATH and/or sys.path

I had two modules, both marked as source roots, with conflicting subfolder names. It could not find modules inside the second folder. Right click → Unload modules helped; I could unload the conflicting module, after this it happily started importing the files from the second source root. This is a band-aid fix but for now it works; TODO later find out a better way to solve these problems.

If not, there are always uglier hacks like sys.path.extend(['folder_marked_as_source_root_that_doesn't_seem_to_get_added_to_$PATH'])

Emulate Python prompt during debugging in Intellij Idea

An interactive shell can be used during debugging in intellij idea! See Interactive console during debug? – IDEs Support (IntelliJ Platform) | JetBrains

Keras saving model ‘Not JSON Serializable’ error

Cannot export Keras model TypeError: (‘Not JSON Serializable:’, b’\n…’) · Issue #27112 · tensorflow/tensorflow - yet another reason to use Keras’ own methods. tf.concat -> tf.keras.layers.Concatenate etc etc etc.

Keras loading weights from only some layers of the model

machine learning - How to load only specific weights on Keras - Stack Overflow

If your first 9 layers are consistently named between your original trained model and the new model, then you can use model.load_weights() with by_name=True. This will update weights only in the layers of your new model that have an identically named layer found in the original trained model.

NLP Reddit

Natural Language Processing exists and is surprisingly active.

Keras Transformer Bert implementation

kpot/keras-transformer: Keras library for building (Universal) Transformers, facilitating BERT and GPT models is a non-official Transformer and Bert implementation using Keras >= 2.0

Stanford CS 224N - Natural Language Processing with Deep Learning

Stanford CS 224N | Natural Language Processing with Deep Learning has lectures, slides etc etc etc. Based on the lecture 06 from 2019, it’s freaking awesome, I really like the way the slides are done. I really really really like the way slides are done. In general that course would answer most of the questions I’ll ever have

It also has literature suggestions, all free to read online. Very glad I found it.

English

A Snowclone is a cliché and phrasal template that can be used and recognized in multiple variants.

Examples:

• Eskimo words for snow
• In space, no one can X
• X is the new Y
• The mother of all X
• X-ing while Y
• To X or not to X
• Have X, will travel
• X considered harmful

Another explanation of Transformer - “The Annotated Transformer”

The Annotated Transformer seems the very best explanation I’ve found. It’s a Jupyter notebook, very detailed and containing an implementation. Link found here: course-nlp/8-translation-transformer.ipynb at master · fastai/course-nlp which in turn is a Jupyter Notebook used in this nice Youtube video lecture: Introduction to the Transformer (NLP video 17) - YouTube.

In this post I present an “annotated” version of the paper in the form of a line-by-line implementation. I have reordered and deleted some sections from the original paper and added comments throughout.

In general everything posted by the Harvard NLP team is very interesting for me especially: Code. It’s all nicely visualized and/or with source code.

Linux watch command

It runs a command continuously and updates the screen when the output changes. Found in my zsh history, watch nvidia-smi is one example.

Heaps’ law

Heaps’ law - Wikipedia “is an empirical law which describes the number of distinct words in a document (or set of documents) as a function of the document length (so called type-token relation)”. In “See also” it has quite a large amound of other “laws” which may be interesting.

unittest @skipUnless Python unit testing

models/transformer_main_test.py at master · tensorflow/models uses a neat thing:

@unittest.skipUnless(tf.test.is_built_with_cuda(), 'requires GPU')
def test_train_1_gpu_with_dist_strat(self):
  FLAGS.distribution_strategy = 'one_device'
  t = transformer_main.TransformerTask(FLAGS)
  t.train()}

Linux keyboard compose key

Yet another change to the layout - now compose lives on rwin: setxkbmap -option -option 'grp:rctrl_toggle, compose:rwin' v4,ru &

This makes the entire file as follows:

setxkbmap -option -option 'grp:rctrl_toggle, compose:rwin' v4,ru &
xrandr --output HDMI-2 --mode 2560x1440 --pos 1920x0 --rotate normal --output HDMI-1 --off --output DP-1 --off --output eDP-1 --primary --mode 1920x1080 --pos 0x0 --rotate normal --output DP-2 --off
xcape -e 'Control_L=Escape' -t 100  &
xmodmap ~/s/mod4 &
keynav &

Python ModuleNotFoundError ‘requests’

To get requests one has to install requests-html. requests might already be installed.

Why did I get ModuleNotFoundError instead of ImportError? Apparently it’s new in Python 3.6 and it’s a subclass of ImportError, just more precise; catching ImportError would work for it too.

Programming cases name

From naming - What are the different kinds of cases? - Stack Overflow:

• Flat case: myvariable
• Kebab case: my-variable (this is the most famous. Other names include: caterpillar case, dash case, hyphen case, lisp case, spinal case and css-case)
• Camel case: myVariable
• Pascal case: MyVariable (other names: capital camel case)
• Snake case: my_variable (other names: c case)
• Macro case: MY_VARIABLE (other names: upper case)
• Cobol case: COBOL-CASE (other names: Train case)

Vim “Done” map/macro

The following inside the .vimrc moves the selected line to the end of the file and takes the cursor back: map <C-d> dGp''

For details, cursor movement - How do I jump to the location of my last edit? - Vi and Vim Stack Exchange.

From the first answer:

The `. command will bring you to your last change.

The ` goes to a mark, and . is a “special” mark which is automatically set to the position where the last change was made. See :help `. for some more information.

There is also `` which will bring you back to where the cursor was before you made your last jump. See :help `` for more information.

Another useful mark is `^; this is the position where the cursor was the last time when insert mode was stopped. See :help `^.

See :help mark-motions for some more general info about using marks (including some other “special” marks that are automatically set).

Tensorflow

tf.math.count_nonzero() works for booleans, as in “number of True elements in tensor”

Python mutable/immutable dictionaries/lists changing inside a function

Tutorial: Why Functions Modify Lists, Dictionaries in Python just got bitten by this again :) TL;DR lists and dicts are mutable, therefore if I pass them to a function and it does stuff to it they will get changed outside the function too. dict.copy() helps.

Adding concat.sh to dtb

One thing which I continuously missed was a way to quickly search through all the files visually - :Ag as fuzzy search is a really nice solution, but I still like the usual way. So now the following was added:

cat * | grep -v "layout: post" | grep -v "categories: \[\]" > ../master_file.md

This is purely a text file that I plan to work with as a text file (though it gets compiled to .html during deployment), and we’ll see what happens next with it.

Random / interesting

Compass rose - Wikipedia, especially the names of the winds. I wonder if they could become names for hostnames/servers or something.

CNN Tutorial - Machine Learning

Simple Introduction to Convolutional Neural Networks is really nice and has pictures of swans.

Taskwarrior .zsh alias current sprint change

Changed zsh alias to alias s='task s sprint.is:$SPRINT or sprint:c', it didn’t have the .is before showing me 40, 41… for sprint 4 etc.

Taskwarrior seq recurring tasks commands

For next year:

10270  for i in $(seq 4 9 52)\nt add project:h +MOD  sprint:$i change toothbrush
10272  for i in $(seq 4 4 52)\nt add project:h +MOD  sprint:$i Master monthly backup