Segmenting

A statistical NLP approach based on supervised learningImage by Gordon Johnson from PixabayIn a previous post on Medium, we discussed segmenting text into sentences . Now we look at a related problem: segmenting text into paragraphs.At first glance, it may seem that the two problems are essentially the same, only at different levels of chunking. The problem of segmenting text into paragraphs is in fact far more interesting.For one thing, sentence boundaries have explicit signals such as periods, question marks, or…

Feature engineering, statistical model, and learning from feedbackImage by Nile from PixabayIn NLP, segmenting a text document into its sentences is a useful basic operation. It is the first step in many NLP tasks that are more elaborate. Such as detecting and correcting errors in the text as it is being written , or detecting named entities .In the former, the idea is that common errors don’t cross sentence boundaries. This holds for the latter as well. Named entities also tend not to cross sentence boundaries.In either…