tree ac5c57af3025d7415b3dd8a5189f9a6ebe6b99f4
parent 8dfffe40186e0bbfd2cabf23b68d880b4a2f62f7
author Ole Rehmsen <oler@google.com> 1627636183 +0200
committer Ole Rehmsen <oler@google.com> 1627636183 +0200

Split tokens using the regexp \w word class

This will then also work for non ASCII characters, such as Umlauts and
other special characters in non-English text.

Change-Id: I6b227724657bb601821f5af205446a0916fba5f8