Search engines can be introduced as a best tool for managing, retrieving and extracting important information from a massive set of web data. These engines are scheduled to search the vast web environment and collect countless pages stored in every corner of the web. Search engines providers are always looking for improving the relationship between the results and reducing response times to users, but both of these can be influenced by the automated traffic sent by the bots. This article first defines bots and challenges of detecting them. Then, it provides a method named ‘boof’ for detecting Search robots. In ‘the boof method’, to achieve high accuracy in detecting anomaly robots, many different parameters are used to model the users’ behavior. After determining the priority of parameters in detecting users, decision tree is made and attempted to categorize users into groups of humans, bots, legal bots and the unknown. Robots detected in the decision tree, enable another part of the robot detection system to identify robots even with low request rate. This is done by detecting the botnet behavior pattern. Evaluation of the proposed method on test data shows 97.7 percent accuracy in recognizing users that this improves the accuracy of at least 9, 9 percent compared to the methods examined previously in this area. This is a significant digit that influences decision-making about 2230 users during each day.

Analysis of published news content is one of the most important issues in information retrieval. Much research has been conducted to analyze individual news articles, while most news events in the media are published in the form of several related articles. Event detection is the task of discovering and grouping documents that describe the same event. It also facilitates better navigation of users in news spaces by presenting an understandable structure of news events. With rapid and increasing growth of online news, the need for search engines to retrieve news events is felt more than ever. The main assumption of event detection is that the words associated with an event appear in the same time windows and similar documents. Accordingly, in this research, we propose a retrospective and feature-pivot method that clusters words into groups according to semantic and temporal features. We then use these words to produce a time frame and a human readable text description. The proposed method is evaluated on the All The News dataset, which consists of two hundred thousand articles from 15 news sources in 2016 and compared to other methods. The evaluation shows that the proposed method outperforms previous methods in terms of precision and recall.

The present investigation concerns evaluation, comparison and analysis of search options existing within web-based meta-search engines. 64 meta-search engines were identified. 19 meta-search engines that were free, accessible and compatible with the objectives of the present study were selected. An author’s constructed check list was used for data collection. Findings indicated that all meta-search engines studied used the AND operator, phrase search, number of results displayed setting, previous search query storage and help tutorials. Nevertheless, none of them demonstrated any search options for hypertext searching and displaying the size of the pages searched. 94.7% support features such as truncation, keywords in title and URL search and text summary display. The checklist used in the study could serve as a model for investigating search options in search engines, digital libraries and other internet search tools.

Today، the importance of text processing and its usages is well known among researchers and students. The amount of textual، documental materials increase day by day. So we need useful ways to save them and retrieve information from these materials. For example، search engines such as Google، Yahoo، Bing and etc. need to read so many web documents and retrieve the most similar ones to the user query. In this example، necessity of real time ability should be mentioned. Keyphrase extraction and some other fields like Information extraction، natural language processing، text summarization، query understanding، machine translation، and text similarity are subsets of text processing. So many efforts in text processing have been established، but there are still many open problems، especially in semantically document understanding subjects. Although these subjects seem not to be very hard for humankind but they are very complex and confusing for a computer، because there is no standard structure to save documents so that computers be able to extract semantics and contents. Document understanding and keyphrase extraction are some of the most important text processing goals. Many statistical and linguistic approaches are proposed in order to address these complex goals. Some methods work based on multi documents and some others on single document which all are generally more difficult than multi documents methods. Some methods use learning algorithms with training data and some others do not. Using natural language processing tools or resources-like ontologies-are effective ways to improve results، but these tools are not reliable for all languages. There are some articles for keyphrase extraction based on co-occurrence and also some statistical methods. Moreover، sometimes it is an important feature for a method to make real time outputs. Based on these characteristics، many approaches have been proposed in the literature. In this paper، we present a new approach for keyphrase extraction from a single document. We present a language-independent approach based on combination of statistical information extracted from document and some logical rules named fundamental text rules. In this approach، there is no need to any natural language processing، nor to ontology and nor to any document corpus. We illustrate a real time method to understand each document focuses by extracting its phrases from segmented document without using any learning algorithm. Then، the Score for each phrase is calculated based on its occurrence and its related phrases occurrences. Then، fundamental text rules omit some phrases based on their scores and their places in text. Remained phrases shows the document focuses. Evaluation shows that our approach takes a high recall and precision in key phrase extraction with very good accuracy in text focuses understanding. These keyphrases extracted of a text presents the most important concepts of that text and it is used to retrieve documents in search engines more efficiently.

Today using the internet has spread wildly, and increasing number of web pages leads to importance of using search engines, therefore some people try to misguide search engines to have more customers and benefit. They increase the rank of their pages by some illegal ways. search engines to. Identify of this kind of web pages can improve search engines and attract confidence to user. According to importance of finding spam pages, the research is presented a new linke-based way to detect spam pages in Persian web graph. This way, first link farms detectes. Finally, the negative scores of spam pages propagate in whole of web graph. This way was implemented on data of Parsijoo search engine and the result of data analyses indicates 21.2% improvement in p@n factor.