discover and index web pages with seo-friendly xml sitemap crawl by search bots. indexed vs non-indexed pages for

indexed nonindexed search bot sitemap crawl seo pages website robots crawler txt engine crawlable xml web media concept database marketing internet relevant social technology optimization information design network online management digital illustration development icon vector research optimize traffic ranking flat internal links content meta tags attributes directives follow nofollow snippets xrobotstag