Open data

New, purely vision-based, segmentation technique is formally described.Only a few simple visual cues are used to assess similarity of the rectangles.Its performance better by an order of magnitude when compared with competition.Rectangle clustering is a viable way to perform web page segmentation. This paper presents a novel approach to web page segmentation, which is one of substantial preprocessing steps when mining data from web documents. Most of the current segmentation methods are based on algorithms that work on a tree representation of web pages (DOM tree or a hierarchical rendering model) and produce another tree structure as an output.In contrast, our method uses a rendering engine to get an image of the web page, takes the smallest rendered elements of that image, performs clustering using a custom algorithm and produces a flat set of segments of a given granularity. For the clustering metrics, we use purely visual properties only: the distance of elements and their visual similarity.We experimentally evaluate the properties of our algorithm by processing 2400web pages. On this set of web pages, we prove that our algorithm is almost 90% faster than the reference algorithm. We also show that our algorithm accuracy is between 47% and 133% of the reference algorithm accuracy with indirect correlation of our algorithms accuracy to the depth of inspected page structure. In our experiments, we also demonstrate the advantages of producing a flat segmentation structure instead of an hierarchy.

Box clustering segmentation

User studies are typically difficult, recruiting enough users is often problematic and each experiment takes a considerable amount of time to be completed. In these studies, eye tracking is increasingly used which often increases time, therefore, the lower the number of users required for these studies the better for making these kinds of studies more practical in terms of economics and time expended. The possibility of achieving almost the same results with fewer users has already been raised. Specifically, the possibility of achieving 75% similarity to the results of 65 users with 27 users for searching tasks and 34 users for browsing tasks has been observed in scanpath trend analysis which discovers the most commonly followed path on a particular web page in terms of its visual elements or areas of interest (AOIs). Different approaches are available to segment or divide web pages into their visual elements or AOIs. In this paper, we investigate whether the possibility raised by the previous work is restricted to a particular page segmentation approach by replicating the experiments with two other segmentation approaches. The results are consistent with ~5% difference for the searching tasks and ~10% difference for the browsing tasks.

/pdf/less-users-more-confidence-how-aois-don-t-affect-scanpath-2c8ccgo2gf.pdf

Less users more confidence: How AOIs don’t affect scanpath trend analysis

Behavior Model Construction for Client Side of Modern Web Applications

Extracting the main content from a web page is essential in various applications such as web crawlers and browser reader modes. Existing extraction methods using text-based algorithms and features for English text can be ineffective for non-English web pages. This study proposes a main content extraction method that obtains visual and structural features from the rendered web page. Our method uses the first impression area (FIA), a part of a web page that users initially view. In this area, websites have applied many techniques that enable users to find the main content easily. Using the non-textual properties in the FIA, our method selects three points with high content area density and expands the area from each point until it meets several structural and visual-based conditions. We evaluated our method, browsers’ (Mozilla Firefox and Google Chrome) reader modes, and existing main content extraction methods on multilingual datasets using two measures: Longest Common Subsequences and matched text blocks. The results showed that our method performed better than other methods in both English (up to 46%, matched text blocks F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.5</sub> ) and non-English (up to 42%, matched text blocks F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.5</sub> ) web pages. 

Extracting the Main Content of Web Pages Using the First Impression Area

Web page segmentation has extensive value in web applications. As the traditional method which only based on Document Object Model (Dom) tree poorly reflects the actual semantic structure of a page, the vision-based measures which attempt to understand the perception of user have attracted great concern in recently. Moreover, the visual page layout structuring is more suitable to suggest a semantic partitioning of a page. Vision-based Page Segmentation (VIPS) algorithm is a notable technique which improves the situation that the visual effect is not consistent with the corresponding DOM tree of web pages. However, due to the increasing complicated structure of web pages and ever-changing of web design, the rules in VIPS become numerous and are no longer fully applicable. To alleviate the shortcoming of VIPS, this paper introduces Hough transform in image processing and takes advantage from DOM trees and visual cues. Our proposed method first extracts the visual separator in web pages according to the perceptive of web designers, then adjusts the segmented information blocks by Hough transformation in the hope of enhancing the VIPS algorithm compatibility and improving the performance of information extraction in applications. The quality of our proposed method is evaluated by subjective and objective measurements, and the experiment results show that this method has come to an anticipant result which even better than the classical one.

Tingting Wei(

Papers

Web page segmentation based on the hough transform and vision cues