<!--noindex--><div class="lwptoc lwptoc-inherit" data-smooth-scroll="1" data-smooth-scroll-offset="24"><div class="lwptoc_i"> <div class="lwptoc_header" style="font-size:110%;">
<b class="lwptoc_title" style="font-weight:900;">Table Of Contents</b> <span class="lwptoc_toggle">
<a class="lwptoc_toggle_label" data-label="show" href="https://getsocialguide.com/robots-txt-file/">hide</a> </span>
</div>
<div class="lwptoc_items lwptoc_items-visible" style="font-size:105%;">
<div class="lwptoc_itemWrap"><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#introduction_to_robotstxt_file_what_is_robotstxt_file">
<span class="lwptoc_item_number">1)</span>
<span class="lwptoc_item_label">Introduction to robots.txt File – What is Robots.txt File?</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#whats_a_robotstxt_file">
<span class="lwptoc_item_number">2)</span>
<span class="lwptoc_item_label">What’s a robots.txt file?</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#instance_of_robotictxt">
<span class="lwptoc_item_number">3)</span>
<span class="lwptoc_item_label">Instance of robotic.txt</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#what_are_search_engines_like_google_and_yahoo_user_agents">
<span class="lwptoc_item_number">4)</span>
<span class="lwptoc_item_label">What are search engines like google and yahoo, user agents?</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#derivative_for_robotstxt_file">
<span class="lwptoc_item_number">5)</span>
<span class="lwptoc_item_label">Derivative for robots.txt file</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#supported_directives_for_robotstxt">
<span class="lwptoc_item_number">6)</span>
<span class="lwptoc_item_label">Supported directives for robots.txt</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#disallow">
<span class="lwptoc_item_number">7)</span>
<span class="lwptoc_item_label">Disallow</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#allow">
<span class="lwptoc_item_number">8)</span>
<span class="lwptoc_item_label">Allow</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#whats_the_significance_of_sitemap_in_robotstxt_file">
<span class="lwptoc_item_number">9)</span>
<span class="lwptoc_item_label">What’s the significance of Sitemap in robots.txt file</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#instance_of_a_robotstxt_file_utilizing_the_sitemap_directive">
<span class="lwptoc_item_number">10)</span>
<span class="lwptoc_item_label">Instance of a robots.txt file utilizing the sitemap directive:</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#the_best_way_to_audit_errors_in_your_robotstxt_file">
<span class="lwptoc_item_number">11)</span>
<span class="lwptoc_item_label">The best way to audit errors in your robots.txt file?</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#how_i_can_create_a_robotstxt_file">
<span class="lwptoc_item_number">12)</span>
<span class="lwptoc_item_label">How I can create a robots.txt file?</span>
</a>
</div><div class="lwptoc_item"> <a href="https://getsocialguide.com/robots-txt-file/#use_a_separate_robotstxt_file_for_every_subdomain">
<span class="lwptoc_item_number">13)</span>
<span class="lwptoc_item_label">Use a separate robots.txt file for every subdomain</span>
</a>
</div></div></div>
</div></div><!--/noindex--><span class="wtr-time-wrap before-content"><span class="wtr-time-number">7</span> min read</span><div data-bg="#f2f5f7" data-comments="" data-commentsbg="#0f54c4" data-content-offset="0" data-endfg="transparent" data-fg="#0d3079" data-fgopacity="1.00" data-location="page" data-mute="1" data-mutedfg="#2348dd" data-mutedopacity="0.84" data-non-touch="1" data-placement="bottom" data-placement-offset="0" data-placement-offset-touch="0" data-placement-touch="top" data-rtl="" data-shadow="1" data-touch="" data-transparent="1" data-width="14" id="wtr-content"><h2><span id="introduction_to_robotstxt_file_what_is_robotstxt_file">Introduction to robots.txt File – What is Robots.txt File?</span></h2>
<header class="entry-header">
<h3><span id="whats_a_robotstxt_file"><span class="synonym">What’s</span> a robots.txt file?</span></h3>
<p>A <span class="synonym">robotic</span>.txt file is used <span class="synonym">to manage</span> <span class="synonym">the best way</span> <span class="synonym">search engines like google and yahoo</span> like google, yahoo, bing, <span class="synonym">and so forth</span> <span class="synonym">Person</span>-agents will <span class="synonym">have the ability to</span> <span class="synonym">entry</span> your <span class="synonym">web site</span> and <span class="synonym">decide</span> if <span class="synonym">they need to</span> be allowed <span class="synonym">to go to</span> or not. The robots.txt file is an integral <span class="synonym">a part of</span> <span class="synonym">internet</span> <span class="synonym">safety</span> and <span class="synonym">ought to be</span> edited when <span class="synonym">wanted</span>.</p>
<h2><span id="instance_of_robotictxt"><span class="synonym">Instance</span> of <span class="synonym">robotic</span>.txt</span></h2>
<p><span class="synonym">It’s</span> <span class="synonym">primarily</span> <span class="synonym">positioned</span> <span class="synonym">within the</span> root folder of your <span class="synonym">web site</span> e.g. <span class="synonym">on this</span> <span class="synonym"><a href="https://getsocialguide.com/how-to-create-a-wordpress-web-site-offline/">web site</a></span> case it <span class="synonym">ought to be</span> like</p>
<p><code>https://getsocialguide.com/<span class="synonym">robotic</span>.txt</code></p>
<p>A typical <span class="synonym">instance</span> of <span class="synonym">robotic</span>.txt file <span class="synonym">should</span> like</p>
<pre><code>Sitemap: https://www.getsocialguide.com/sitemap.xml
<span class="synonym">User</span>-agent: *
Disallow: /wp-admin/
<span class="synonym">Allow</span>: /wp-admin/admin-ajax.php</code></pre>
<h2><span id="what_are_search_engines_like_google_and_yahoo_user_agents">What are <span class="synonym">search engines like google and yahoo</span>, <span class="synonym">user</span> <span class="synonym">agents</span>?</span></h2>
<p>When a <a href="https://getsocialguide.com/search-engine-optimization-services/">search engine</a> bot crawls your <span class="synonym">web site</span> <span class="synonym">each</span> <a href="https://getsocialguide.com/plus-100-free-blog-directories-to-submit-your-blog/">search engine</a> bot identifies themselves with <span class="synonym">particular</span> user-agent string. <span class="synonym">You may</span> set the <span class="synonym">customized</span> <span class="synonym">directions</span> in your robots.txt file for <span class="synonym">every</span> <span class="synonym">of those</span>. <span class="synonym">Tons of</span> of user-agents exist, <span class="synonym">however</span> <span class="synonym">the next</span> are some <span class="synonym">helpful</span> ones for <span class="synonym"><a href="https://getsocialguide.com/what-search-engine-optimization-stands-for-your-business/">search engine</a> marketing</span>:</p>
<ul>
<li><strong>Google: </strong>Googlebot</li>
<li><strong>Google <span class="synonym">Pictures</span>:</strong> Googlebot-<span class="synonym">Picture</span></li>
<li><strong>Yahoo: </strong>Slurp</li>
<li><strong>Bing: </strong>Bingbot</li>
<li><strong>DuckDuckGo:</strong> DuckDuckBot</li>
<li><strong>Baidu</strong>: Baiduspider</li>
</ul>
</header>
<header class="entry-header">
<h2><span id="derivative_for_robotstxt_file"><span class="synonym">Derivative</span> for robots.txt file</span></h2>
<p><span class="synonym">You’ll be able to</span> <span class="synonym">management</span> <span class="synonym">by way of</span> robots.txt file how <span class="synonym">every</span> <span class="synonym">of those</span> user-agents crawls your <span class="synonym">web site</span>. <span class="synonym">There are many</span> spam bot user-agents which <span class="synonym">eat</span> <span class="synonym">plenty of</span> your <span class="synonym">web site</span> bandwidth by <span class="synonym">robotic</span>.txt file <span class="synonym">you possibly can</span> block them, <span class="synonym">for instance</span> <span class="synonym">for instance</span> you <span class="synonym">need to</span> <span class="synonym">permit</span> <span class="synonym">solely</span> Googlebot to crawl your <span class="synonym">web site</span> and block all <span class="synonym">different</span> bots from doing so, for this, <span class="synonym">you should utilize</span> <span class="synonym">the next</span> code in your robots.txt file.</p>
<p><strong><span class="synonym">Instance</span> of Disallow <span class="synonym">different</span> bots</strong></p>
<pre><code><span class="synonym">Person</span>-agent: *
Disallow: /
<span class="synonym">Person</span>-agent: Googlebot
<span class="synonym">Enable</span>:</code> /</pre>
<p><span class="synonym">Needless to say</span> as many user-agents as <span class="synonym">you want</span>, your robots.txt file can <span class="synonym">embody</span> directives. That being <span class="synonym">mentioned</span>, it acts as a <span class="synonym">clear</span> slate <span class="synonym">every</span> time you declare <span class="synonym">a brand new</span> user-agent. In <span class="synonym">different</span> <span class="synonym">phrases</span>, <span class="synonym">including</span> directives for <span class="synonym">a number of</span> user-agents <span class="synonym">doesn’t</span> apply the directives declared for <span class="synonym">the primary</span> user-agent to the second, or third, or fourth, <span class="synonym">and so forth</span>.</p>
<h2><span id="supported_directives_for_robotstxt">Supported directives for robots.txt</span></h2>
<p>Following is the <span class="synonym">checklist</span> of some <span class="synonym">primary</span> directives that Googlebot <span class="synonym">at present</span> <span class="synonym">helps</span>.</p>
<h3><span id="disallow">Disallow</span></h3>
<p>Use this directive to instruct <span class="synonym"><a href="https://getsocialguide.com/how-to-rank-on-first-page-of-google/">search engines</a> like google</span> <span class="synonym">to not</span> <span class="synonym">entry</span> <span class="synonym">recordsdata</span> and pages falling <span class="synonym">inside</span> <span class="synonym">a specific</span> path. <span class="synonym">For instance</span>, <span class="synonym">if you happen to</span> <span class="synonym">wished</span> <span class="synonym">to dam</span> all <span class="synonym">the major <a href="https://getsocialguide.com/what-search-engine-optimization-stands-for-your-business/">search engines</a></span> from accessing your <span class="synonym">weblog</span> and all its posts, this <span class="synonym">would possibly</span> <span class="synonym">seem like</span> your robots.txt file:</p>
<pre><code><span class="synonym">User</span>-agent: *
Disallow: /<span class="synonym"><a href="https://getsocialguide.com/blog/">blog</a></span></code></pre>
<h3><span id="allow"><span class="synonym">Allow</span></span></h3>
<p>Use this directive to let <span class="synonym"><a href="https://getsocialguide.com/what-search-engine-optimization-stands-for-your-business/">search engines</a> like google</span> crawl a subdirectory or <span class="synonym">web page</span> — even in a <span class="synonym">listing</span> <span class="synonym">in any other case</span> disallowed. <span class="synonym">For instance</span>, <span class="synonym">if you happen to</span> <span class="synonym">did not</span> <span class="synonym">need</span> <span class="synonym"><a href="https://getsocialguide.com/how-to-rank-on-first-page-of-google/">search engines like google</a></span> to <span class="synonym">entry</span> any <span class="synonym">publish</span> <span class="synonym">in your</span> <span class="synonym">weblog</span> <span class="synonym">besides</span> one, your robots.txt file <span class="synonym">would possibly</span> <span class="synonym">seem like</span> this:</p>
<pre><code><span class="synonym">User</span>-agent: *
Disallow: /myblog
<span class="synonym">Allow</span>: /myblog/allowed-pos/</code></pre>
<p><span class="synonym">Within the</span> above <span class="synonym">instance</span>, a search engine can <span class="synonym">access</span> <code>/myblog/allowed-post/</code>. <span class="synonym">However</span> <span class="synonym">they can</span>’t <span class="synonym">access</span>:</p>
<div style="clear:both; margin-top:0em; margin-bottom:0em;"><a class="u95d8482e0cc7f445aeeb906e3d1d1c47" href="https://getsocialguide.com/strengthen-your-wordpress-with-these-security-tips/" rel="nofollow" target="_blank"><!-- INLINE RELATED POSTS 1/2 //--><style> .u95d8482e0cc7f445aeeb906e3d1d1c47 { padding:0px; margin: 0; padding-top:1em!important; padding-bottom:1em!important; width:100%; display: block; font-weight:bold; background-color:inherit; border:0!important; border-left:4px solid #34495E!important; text-decoration:none; } .u95d8482e0cc7f445aeeb906e3d1d1c47:active, .u95d8482e0cc7f445aeeb906e3d1d1c47:hover { opacity: 1; transition: opacity 250ms; webkit-transition: opacity 250ms; text-decoration:none; } .u95d8482e0cc7f445aeeb906e3d1d1c47 { transition: background-color 250ms; webkit-transition: background-color 250ms; opacity: 1; transition: opacity 250ms; webkit-transition: opacity 250ms; } .u95d8482e0cc7f445aeeb906e3d1d1c47 .ctaText { font-weight:bold; color:inherit; text-decoration:none; font-size: 16px; } .u95d8482e0cc7f445aeeb906e3d1d1c47 .postTitle { color:#3498DB; text-decoration: underline!important; font-size: 16px; } .u95d8482e0cc7f445aeeb906e3d1d1c47:hover .postTitle { text-decoration: underline!important; } </style><div style="padding-left:1em; padding-right:1em;"><span class="ctaText">Related Post</span> <span class="postTitle">Strengthen your WordPress with these Security Tips</span></div></a></div><p><code>/<span class="synonym">weblog</span>/notallowed-post/</code><br/>
<code>/<span class="synonym">weblog</span>/notallowed2-post/</code><br/>
<code>/<span class="synonym">weblog</span>/notallowed3.pdf/</code></p>
<p><span class="synonym">Each</span> Google and Bing <span class="synonym"><a href="https://getsocialguide.com/what-search-engine-optimization-stands-for-your-business/">search engines</a> like google</span> <span class="synonym">assist</span> this directive.</p>
<h2><span id="whats_the_significance_of_sitemap_in_robotstxt_file"><span class="synonym">What’s the</span> <span class="synonym">significance</span> of Sitemap in robots.txt file</span></h2>
<p>Use this directive to specify <span class="synonym">the situation</span> of <span class="synonym">the major search engines</span> in your sitemap(s). <span class="synonym">In case you are</span> unfamiliar with sitemaps, <span class="synonym">they typically</span> <span class="synonym">embody</span> the pages you <span class="synonym">want to</span> crawl and index <span class="synonym">search engines like google</span>.</p>
<h3><span id="instance_of_a_robotstxt_file_utilizing_the_sitemap_directive"><span class="synonym">Instance</span> of a robots.txt file <span class="synonym">utilizing</span> the sitemap directive:</span></h3>
<pre><code>Sitemap: https://www.mydomain.com/sitemap.xml
<span class="synonym">User</span>-agent: *
Disallow: /myblog/
<span class="synonym">Allow</span>: /myblog/<span class="synonym">what is</span> <span class="synonym">importance</span> of robots.txt file/</code></pre>
<p>How <span class="synonym">necessary</span> is to have your sitemap(s) included in your robots.txt file? <span class="synonym">In the event you</span>‘ve already submitted <span class="synonym">by</span> Search Console, then Google will <span class="synonym">discover</span> it <span class="synonym">considerably</span> redundant. It does <span class="synonym">inform</span> <span class="synonym">different</span> <span class="synonym"><a href="https://getsocialguide.com/how-to-rank-on-first-page-of-google/">search engines like google</a></span> <span class="synonym">resembling</span> Bing <span class="synonym">the place</span> <span class="synonym">to seek out</span> your sitemap, <span class="synonym">nonetheless</span>, so <span class="synonym">it is</span> <span class="synonym">nonetheless</span> good <span class="synonym">apply</span>. <span class="synonym">Word</span> that for <span class="synonym">every</span> user-agent <span class="synonym">you needn’t</span> repeat the sitemap directive <span class="synonym">a number of</span> <span class="synonym">occasions</span>. It <span class="synonym">doesn’t</span> apply to <span class="synonym">only one</span> <span class="synonym">individual</span>. So, <span class="synonym">at the start</span> or <span class="synonym">finish</span> of your robots.txt file, you’d <span class="synonym">higher</span> <span class="synonym">embody</span> sitemap directives. e.g</p>
<pre><code>Sitemap: https://www.mydomain.com/sitemap.xml
<span class="synonym">User</span>-agent: Googlebot
Disallow: /myblog/
<span class="synonym">Allow</span>: /myblog/<span class="synonym">what is</span> <span class="synonym">significance</span> of robots.txt file/
<span class="synonym">User</span>-agent: Bingbot
Disallow: /downloads/</code></pre>
<h2><span id="the_best_way_to_audit_errors_in_your_robotstxt_file"><span class="synonym">The best way to</span> audit errors in your robots.txt file?</span></h2>
<p>Robots.txt errors can <span class="synonym">comparatively</span> <span class="synonym">simply</span> slip <span class="synonym">by</span> <span class="synonym">the online</span>, so it pays <span class="synonym">to maintain</span> <span class="synonym">a watch</span> out for <span class="synonym">issues</span>.</p>
<p> </p>
<p><span class="synonym">To do that</span>, <span class="synonym">examine</span> the “<span class="synonym">Protection</span>” report <span class="synonym">within the</span> <a href="https://search.google.com/search-console/" rel="noopener" target="_blank">Google </a>Search Console <span class="synonym">often</span> for <span class="synonym">points</span> <span class="synonym">associated</span> to robots.txt. <span class="synonym">Under</span> are <span class="synonym">among the</span> <span class="synonym">errors</span> <span class="synonym">that you simply</span> <span class="synonym">would possibly</span> see, what they <span class="synonym">imply</span> <span class="synonym">and the way</span> <span class="synonym">you may</span> <span class="synonym">repair</span> them. Or <span class="synonym">you should utilize</span> any of free <span class="synonym">on-line</span> <span class="synonym">instruments</span> to <span class="synonym">examine</span> robots.txt file errors, Personly <span class="synonym">I’ll</span> <span class="synonym">suggest</span> to <span class="synonym">you utilize</span> <a href="https://technicalseo.com/tools/robots-txt/" rel="noopener" target="_blank">https://technicalseo.com/tools/robots-txt</a>/ which <span class="synonym">permits</span> to crawl your <span class="synonym">web site</span> and <span class="synonym">examine</span> if there any errors exit in robots.txt file.</p>
<h2><span id="how_i_can_create_a_robotstxt_file">How I can create a robots.txt file?</span></h2>
<p><span class="synonym">When you’ve got</span> <span class="synonym">discovered</span> that <span class="synonym">you do not have</span> a robots.txt file or <span class="synonym">need to</span> alter yours, <span class="synonym">it is a</span> <span class="synonym">easy</span> <span class="synonym">course of</span> to create one. This <a href="https://support.google.com/webmasters/answer/6062596?hl=en" rel="noopener" target="_blank">Google article </a>goes <span class="synonym">by</span> the creation <span class="synonym">technique of</span> robots.txt <span class="synonym">recordsdata</span>, and <a href="https://support.google.com/webmasters/answer/6062598?hl=en&ref_topic=6061961" rel="noopener" target="_blank">this tool </a><span class="synonym">permits you to</span> <span class="synonym">take a look at</span> <span class="synonym">whether or not</span> your file <span class="synonym">is ready</span> up <span class="synonym">accurately</span>.</p>
<p><span class="synonym">If you end up</span> updating the robots.txt file <span class="synonym">it’s best to</span> <span class="synonym">comply with</span> <span class="synonym">sure</span> steps. You <span class="synonym">need to</span> <span class="synonym">be sure that</span> the file <span class="synonym">you’re</span> <span class="synonym">modifying</span> <span class="synonym">is strictly</span> <span class="synonym">the identical</span> <span class="synonym">because the</span> one <span class="synonym">you have already got</span>. <span class="synonym">Utilizing</span> <span class="synonym">a special</span> <span class="synonym">model</span> of the file <span class="synonym">might</span> confuse <span class="synonym">the major search engines</span> and <span class="synonym">go away</span> them <span class="synonym">questioning</span> what <span class="synonym">the unique</span> file is.</p>
<p><span class="synonym">To start out</span>, <span class="synonym">it’s worthwhile to</span> <span class="synonym">search for</span> <span class="synonym">the present</span> <span class="synonym">model</span> of the file <span class="synonym">and duplicate</span> <span class="synonym">the next</span> <span class="synonym">info</span> into your new <span class="synonym">model</span>. <span class="synonym">Be sure you</span> <span class="synonym">at all times</span> change the extension of the <span class="synonym">textual content</span>. Your <span class="synonym">software program</span> will <span class="synonym">deal with</span> this for you <span class="synonym">mechanically</span>.</p>
<p><span class="synonym">Subsequent</span>, you <span class="synonym">need to</span> <span class="synonym">discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> change. <span class="synonym">You are able to do</span> this by right-clicking on the <span class="synonym">textual content</span> you <span class="synonym">need to</span> change <span class="synonym">and selecting</span> “Go-To” in your <span class="synonym">textual content</span> editor. <span class="synonym">You will note</span> the <span class="synonym">textual content</span> displayed in <span class="synonym">a special</span> editor window. Change the <span class="synonym">textual content</span> to <span class="synonym">the shape</span> <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> edit.</p>
<div style="clear:both; margin-top:0em; margin-bottom:0em;"><a class="u82521326594fbb6b185618f5fb8ae8d9" href="https://getsocialguide.com/free-magazine-wordpress-themes-templates/" rel="nofollow" target="_blank"><!-- INLINE RELATED POSTS 2/2 //--><style> .u82521326594fbb6b185618f5fb8ae8d9 { padding:0px; margin: 0; padding-top:1em!important; padding-bottom:1em!important; width:100%; display: block; font-weight:bold; background-color:inherit; border:0!important; border-left:4px solid #34495E!important; text-decoration:none; } .u82521326594fbb6b185618f5fb8ae8d9:active, .u82521326594fbb6b185618f5fb8ae8d9:hover { opacity: 1; transition: opacity 250ms; webkit-transition: opacity 250ms; text-decoration:none; } .u82521326594fbb6b185618f5fb8ae8d9 { transition: background-color 250ms; webkit-transition: background-color 250ms; opacity: 1; transition: opacity 250ms; webkit-transition: opacity 250ms; } .u82521326594fbb6b185618f5fb8ae8d9 .ctaText { font-weight:bold; color:inherit; text-decoration:none; font-size: 16px; } .u82521326594fbb6b185618f5fb8ae8d9 .postTitle { color:#3498DB; text-decoration: underline!important; font-size: 16px; } .u82521326594fbb6b185618f5fb8ae8d9:hover .postTitle { text-decoration: underline!important; } </style><div style="padding-left:1em; padding-right:1em;"><span class="ctaText">Related Post</span> <span class="postTitle">Free Magazine WordPress Themes & Templates</span></div></a></div><p><span class="synonym">You need to</span> <span class="synonym">discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> change <span class="synonym">and duplicate</span> the <span class="synonym">textual content</span> to your clipboard. Then use your <span class="synonym">textual content</span> editor <span class="synonym">so as to add</span> <span class="synonym">the next</span> into <span class="synonym">the top</span> of the <span class="synonym">textual content</span>. <span class="synonym">Don’t</span> <span class="synonym">kind</span> the extension into the <span class="synonym">textual content</span> <span class="synonym">field</span> <span class="synonym">as a result of</span> <span class="synonym">it is going to</span> change the file <span class="synonym">identify</span>, <span class="synonym">and also you</span> <span class="synonym">will be unable</span> to <span class="synonym">add</span> the file to your <span class="synonym">web site</span>. The extension <span class="synonym">is simply</span> <span class="synonym">necessary</span> <span class="synonym">if you wish to</span> <span class="synonym">add</span> the file later.</p>
<p><img alt="Introduction to robots.txt file for Blogger's" class="n3VNCb aligncenter" data-noaft="1" decoding="async" fetchpriority="high" height="418" src="https://getsocialguide.com/wp-content/uploads/2020/09/07042020-G3D80KGOG6VYTLME9N4Wwhat-is-robots-txt-file.jpg" title="Introduction to robots.txt file for Blogger's 2" width="848"/></p>
<p>After the <span class="synonym">textual content</span> has been added, <span class="synonym">it’s best to</span> <span class="synonym">discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> edit. <span class="synonym">Discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> edit <span class="synonym">and duplicate</span> the <span class="synonym">textual content</span> to your clipboard. <span class="synonym">Subsequent</span>, you <span class="synonym">need to</span> <span class="synonym">discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> change. <span class="synonym">You are able to do</span> this by right-clicking on the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> change <span class="synonym">and selecting</span> “Go-To” in your <span class="synonym">textual content</span> editor. <span class="synonym">You will note</span> the <span class="synonym">textual content</span> displayed in <span class="synonym">a special</span> editor window.</p>
<p><span class="synonym">You need to</span> add the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> change. <span class="synonym">Don’t</span> <span class="synonym">kind</span> the extension into the <span class="synonym">textual content</span> <span class="synonym">field</span> <span class="synonym">as a result of</span> <span class="synonym">it is going to</span> change the file <span class="synonym">identify</span>, <span class="synonym">and also you</span> <span class="synonym">will be unable</span> to <span class="synonym">add</span> the file to your <span class="synonym">web site</span>. The extension <span class="synonym">is simply</span> <span class="synonym">necessary</span> <span class="synonym">if you wish to</span> <span class="synonym">add</span> the file later. After the <span class="synonym">textual content</span> has been added, <span class="synonym">it’s best to</span> <span class="synonym">discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> edit. <span class="synonym">Discover</span> the <span class="synonym">textual content</span> <span class="synonym">that you simply</span> <span class="synonym">need to</span> edit <span class="synonym">and duplicate</span> the <span class="synonym">textual content</span> to your clipboard.</p>
<p><span class="synonym">If you end up</span> <span class="synonym">completed</span> with the robots.txt file <span class="synonym">you possibly can</span> <span class="synonym">take a look at</span> <span class="synonym">the ultimate</span> <span class="synonym">model</span> <span class="synonym">that you’ve got</span> written. Now that <span class="synonym">you understand how</span> to <span class="synonym">replace</span> the robots.txt file, <span class="synonym">it’s best to</span> <span class="synonym">at all times</span> double-check your edits. <span class="synonym">In case you are</span> <span class="synonym">uncertain</span> of <span class="synonym">easy methods to</span> do <span class="synonym">one thing</span>, <span class="synonym">it’s best to</span> <span class="synonym">ship</span> an <span class="synonym">electronic mail</span> to <span class="synonym">the web site</span> <span class="synonym">proprietor</span> <span class="synonym">to seek out</span> out.</p>
<h2><span id="use_a_separate_robotstxt_file_for_every_subdomain">Use a separate robots.txt file for <span class="synonym">every</span> subdomain</span></h2>
<p>Robots.txt <span class="synonym">solely</span> controls crawling <span class="synonym"><a href="https://getsocialguide.com/6-habits-that-will-improve-your-wealth-success-and-productivity/">habits</a></span> on the subdomain <span class="synonym">the place</span> <span class="synonym">it is</span> hosted. <span class="synonym">If you wish to</span> <span class="synonym">management</span> crawling on <span class="synonym">a special</span> subdomain, <span class="synonym">you will</span> <span class="synonym">want</span> a separate robots.txt file. <span class="synonym">For instance</span>, <span class="synonym">in case your</span> <span class="synonym">primary</span> <span class="synonym">web site</span> is <span class="synonym">positioned</span> on abcdomain.com, and your <span class="synonym">weblog</span> is <span class="synonym">positioned</span> on <span class="synonym">weblog</span>.abcdomain.com, you’d <span class="synonym">want</span> two <span class="synonym">files</span> on robots.txt. One <span class="synonym">ought to</span> go into <span class="synonym">the principle</span> <span class="synonym">area</span>‘s root <span class="synonym">listing</span>, and <span class="synonym">one other</span> <span class="synonym">ought to</span> go into the <span class="synonym">weblog</span>‘s root <span class="synonym">listing</span>.</p>
</header>
<p> </p>
<p> </p>
</div><div class="essb_links essb_displayed_bottom essb_share essb_template_default4-retina essb_template_glow-retina essb_1942413643 essb_sharebtn_counter_hidden print-no" data-essb-button-style="button" data-essb-counter-pos="hidden" data-essb-fullurl="https://getsocialguide.com/robots-txt-file/" data-essb-instance="1942413643" data-essb-position="bottom" data-essb-postid="8992" data-essb-template="default4-retina essb_template_glow-retina" data-essb-url="https://getsocialguide.com/robots-txt-file/" id="essb_displayed_bottom_1942413643"><ul class="essb_links_list"><li class="essb_item essb_link_facebook nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-facebook essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F&t=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs" onclick="essb.window('https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F&t=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs','facebook','1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Facebook"><span class="essb_icon essb_icon_facebook"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-facebook" focusable="false" height="16" role="img" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path d="M16 8.049c0-4.446-3.582-8.05-8-8.05C3.58 0-.002 3.603-.002 8.05c0 4.017 2.926 7.347 6.75 7.951v-5.625h-2.03V8.05H6.75V6.275c0-2.017 1.195-3.131 3.022-3.131.876 0 1.791.157 1.791.157v1.98h-1.009c-.993 0-1.303.621-1.303 1.258v1.51h2.218l-.354 2.326H9.25V16c3.824-.604 6.75-3.934 6.75-7.951"></path></svg></span><span class="essb_network_name">Facebook</span></a></li><li class="essb_item essb_link_twitter nolightbox essb_link_twitter_x essb_link_svg_icon"> <a class="nolightbox essb-s-bg-twitter essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://x.com/intent/post?text=Introduction%20to%20robots.txt%20file%20for%20Blogger%27s&url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F" onclick="essb.window('https://x.com/intent/post?text=Introduction%20to%20robots.txt%20file%20for%20Blogger%27s&url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F','twitter','1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Twitter"><span class="essb_icon essb_icon_twitter_x"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-twitter_x" focusable="false" role="img" viewbox="0 0 24 24"><path d="M18.244 2.25h3.308l-7.227 8.26 8.502 11.24H16.17l-5.214-6.817L4.99 21.75H1.68l7.73-8.835L1.254 2.25H8.08l4.713 6.231zm-1.161 17.52h1.833L7.084 4.126H5.117z"></path></svg></span><span class="essb_network_name">Twitter</span></a></li><li class="essb_item essb_link_pinterest nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-pinterest essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://getsocialguide.com/robots-txt-file/" onclick="essb.pinterest_picker('1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Pinterest"><span class="essb_icon essb_icon_pinterest"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-pinterest" focusable="false" height="32" role="img" viewbox="0 0 32 32" width="32" xmlns="http://www.w3.org/2000/svg"><path d="M10.625 12.25c0-1.375 0.313-2.5 1.063-3.438 0.688-0.938 1.563-1.438 2.563-1.438 0.813 0 1.438 0.25 1.875 0.813s0.688 1.25 0.688 2.063c0 0.5-0.125 1.125-0.313 1.813-0.188 0.75-0.375 1.625-0.688 2.563-0.313 1-0.563 1.75-0.688 2.313-0.25 1-0.063 1.875 0.563 2.625 0.625 0.688 1.438 1.063 2.438 1.063 1.75 0 3.188-1 4.313-2.938 1.125-2 1.688-4.375 1.688-7.188 0-2.125-0.688-3.875-2.063-5.25-1.375-1.313-3.313-2-5.813-2-2.813 0-5.063 0.875-6.75 2.688-1.75 1.75-2.625 3.875-2.625 6.375 0 1.5 0.438 2.75 1.25 3.75 0.313 0.313 0.375 0.688 0.313 1.063-0.125 0.313-0.25 0.813-0.375 1.5-0.063 0.25-0.188 0.438-0.375 0.5s-0.375 0.063-0.563 0c-1.313-0.563-2.25-1.438-2.938-2.75s-1-2.813-1-4.5c0-1.125 0.188-2.188 0.563-3.313s0.875-2.188 1.625-3.188c0.75-1.063 1.688-1.938 2.688-2.75 1.063-0.813 2.313-1.438 3.875-1.938 1.5-0.438 3.125-0.688 4.813-0.688 1.813 0 3.438 0.313 4.938 0.938 1.5 0.563 2.813 1.375 3.813 2.375 1.063 1.063 1.813 2.188 2.438 3.5 0.563 1.313 0.875 2.688 0.875 4.063 0 3.75-0.938 6.875-2.875 9.313-1.938 2.5-4.375 3.688-7.375 3.688-1 0-1.938-0.188-2.813-0.688-0.875-0.438-1.5-1-1.875-1.688-0.688 2.938-1.125 4.688-1.313 5.25-0.375 1.438-1.25 3.188-2.688 5.25h-1.313c-0.25-2.563-0.188-4.688 0.188-6.375l2.438-10.313c-0.375-0.813-0.563-1.813-0.563-3.063z"></path></svg></span><span class="essb_network_name">Pinterest</span></a></li><li class="essb_item essb_link_linkedin nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-linkedin essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://www.linkedin.com/shareArticle?mini=true&ro=true&trk=EasySocialShareButtons&title=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs&url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F" onclick="essb.window('https://www.linkedin.com/shareArticle?mini=true&ro=true&trk=EasySocialShareButtons&title=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs&url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F','linkedin','1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on LinkedIn"><span class="essb_icon essb_icon_linkedin"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-linkedin" focusable="false" height="32" role="img" viewbox="0 0 32 32" width="32" xmlns="http://www.w3.org/2000/svg"><path d="M23.938 16.375c-0.438-0.5-1.188-0.75-2.25-0.75-1.375 0-2.313 0.438-2.875 1.188-0.563 0.813-0.813 1.938-0.813 3.375v10.938c0 0.188-0.063 0.375-0.25 0.563-0.125 0.125-0.375 0.25-0.563 0.25h-5.75c-0.188 0-0.375-0.125-0.563-0.25-0.125-0.188-0.25-0.375-0.25-0.563v-20.188c0-0.188 0.125-0.375 0.25-0.563 0.188-0.125 0.375-0.25 0.563-0.25h5.563c0.188 0 0.375 0.063 0.5 0.125s0.188 0.25 0.25 0.438c0 0.188 0.063 0.375 0.063 0.438s0 0.25 0 0.5c1.438-1.375 3.375-2 5.688-2 2.688 0 4.75 0.625 6.25 1.938 1.5 1.375 2.25 3.313 2.25 5.938v13.625c0 0.188-0.063 0.375-0.25 0.563-0.125 0.125-0.375 0.25-0.563 0.25h-5.875c-0.188 0-0.375-0.125-0.563-0.25-0.125-0.188-0.25-0.375-0.25-0.563v-12.313c0-1.063-0.188-1.875-0.563-2.438zM6.625 6.688c-0.75 0.75-1.688 1.125-2.75 1.125s-2-0.375-2.75-1.125c-0.75-0.75-1.125-1.625-1.125-2.75 0-1.063 0.375-1.938 1.125-2.688s1.688-1.188 2.75-1.188c1.063 0 2 0.438 2.75 1.188s1.125 1.625 1.125 2.688c0 1.125-0.375 2-1.125 2.75zM7.563 10.938v20.188c0 0.188-0.125 0.375-0.25 0.563-0.188 0.125-0.375 0.25-0.563 0.25h-5.75c-0.188 0-0.375-0.125-0.563-0.25-0.125-0.188-0.25-0.375-0.25-0.563v-20.188c0-0.188 0.125-0.375 0.25-0.563 0.188-0.125 0.375-0.25 0.563-0.25h5.75c0.188 0 0.375 0.125 0.563 0.25 0.125 0.188 0.25 0.375 0.25 0.563z"></path></svg></span><span class="essb_network_name">LinkedIn</span></a></li><li class="essb_item essb_link_buffer nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-buffer essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://bufferapp.com/add?url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F&text=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs&via=&picture=&count=horizontal&source=button" onclick="essb.window('https://bufferapp.com/add?url=https%3A%2F%2Fgetsocialguide.com%2Frobots-txt-file%2F&text=Introduction+to+robots.txt+file+for+Blogger%5C%26%23039%3Bs&via=&picture=&count=horizontal&source=button','buffer','1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Buffer"><span class="essb_icon essb_icon_buffer"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-buffer" focusable="false" height="32" role="img" viewbox="0 0 32 32" width="32" xmlns="http://www.w3.org/2000/svg"><path d="M31.438 24.313c-0.063-0.063-0.188-0.125-0.25-0.188-0.813-0.375-1.688-0.75-2.5-1.125-0.688-0.375-1.438-0.375-2.188 0-3.125 1.438-6.25 2.938-9.375 4.375-0.75 0.313-1.5 0.375-2.188 0-3.125-1.438-6.313-2.938-9.438-4.375-0.75-0.375-1.5-0.375-2.25 0s-1.563 0.75-2.313 1.063c-0.25 0.125-0.625 0.25-0.625 0.625 0 0.313 0.375 0.375 0.625 0.5 4.625 2.188 9.25 4.375 13.938 6.5 0.75 0.375 1.5 0.375 2.188 0.063 4.688-2.188 9.375-4.375 14.063-6.563 0.125-0.063 0.25-0.125 0.313-0.188 0.313-0.188 0.313-0.5 0-0.688zM0.5 7.688c0.125 0.063 0.313 0.188 0.438 0.25 4.688 2.125 9.313 4.313 13.938 6.5 0.75 0.313 1.5 0.313 2.25 0 4.625-2.188 9.313-4.375 13.938-6.5 0.125-0.125 0.313-0.188 0.438-0.313 0.25-0.125 0.25-0.375 0-0.563-0.125-0.125-0.313-0.188-0.438-0.313-4.625-2.125-9.25-4.313-13.875-6.438-0.813-0.375-1.563-0.375-2.375 0-4.625 2.125-9.25 4.313-13.875 6.5-0.188 0.063-0.313 0.125-0.438 0.188-0.25 0.188-0.25 0.5 0 0.688zM31.438 15.688c-0.063-0.063-0.188-0.125-0.25-0.188-0.813-0.375-1.688-0.75-2.5-1.125-0.688-0.375-1.438-0.375-2.188 0-3.125 1.438-6.25 2.938-9.375 4.375-0.75 0.313-1.5 0.313-2.188 0-3.125-1.438-6.313-2.938-9.438-4.375-0.75-0.375-1.5-0.375-2.25 0s-1.563 0.75-2.313 1.063c-0.25 0.125-0.625 0.25-0.625 0.563 0 0.375 0.375 0.438 0.625 0.563 4.625 2.188 9.25 4.375 13.938 6.5 0.75 0.375 1.5 0.375 2.188 0.063 4.688-2.188 9.375-4.375 14.063-6.563 0.125-0.063 0.25-0.125 0.313-0.188 0.313-0.188 0.313-0.5 0-0.688z"></path></svg></span><span class="essb_network_name">Buffer</span></a></li><li class="essb_item essb_link_love nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-love essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://getsocialguide.com/robots-txt-file/" onclick="essb.loveThis('1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Love This"><span class="essb_icon essb_icon_love"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-love" focusable="false" height="32" role="img" viewbox="0 0 37 32" width="37" xmlns="http://www.w3.org/2000/svg"><path d="M33.2,2.2c-3.9-3.3-9.7-2.7-13.3,1l-1.4,1.4l-1.4-1.4c-3.6-3.7-9.4-4.3-13.3-1C-0.7,6-1,12.9,3.1,17l13.8,14.3c0.9,0.9,2.3,0.9,3.2,0L33.9,17C38,12.9,37.7,6,33.2,2.2L33.2,2.2z"></path></svg></span><span class="essb_network_name">Love This</span></a></li><li class="essb_item essb_link_sharebtn nolightbox essb_link_svg_icon"> <a class="nolightbox essb-s-bg-share essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://getsocialguide.com/robots-txt-file/" onclick="essb.toggle_more('1942413643'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Share"><span class="essb_icon essb_icon_share"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-share" focusable="false" height="32" role="img" viewbox="0 0 32 32" width="32" xmlns="http://www.w3.org/2000/svg"><path d="M25.4,18.7c1.8,0,3.4,0.6,4.7,1.9c1.3,1.3,1.9,2.9,1.9,4.7c0,1.8-0.6,3.4-1.9,4.7c-1.3,1.3-2.9,1.9-4.7,1.9c-1.8,0-3.4-0.6-4.7-1.9c-1.3-1.3-1.9-2.9-1.9-4.7c0-0.1,0-0.4,0-0.7l-7.5-3.7c-1.2,1.2-2.8,1.8-4.5,1.8c-1.8,0-3.4-0.6-4.7-1.9C0.6,19.4,0,17.8,0,16s0.6-3.4,1.9-4.7s2.9-1.9,4.7-1.9c1.8,0,3.3,0.6,4.5,1.8l7.5-3.7c0-0.3,0-0.6,0-0.7c0-1.8,0.6-3.4,1.9-4.7C21.9,0.6,23.5,0,25.4,0c1.8,0,3.4,0.6,4.7,1.9C31.4,3.2,32,4.8,32,6.6s-0.6,3.4-1.9,4.7c-1.3,1.3-2.9,1.9-4.7,1.9c-1.8,0-3.3-0.6-4.5-1.8l-7.5,3.7c0,0.3,0,0.6,0,0.7c0,0.1,0,0.4,0,0.7l7.5,3.7C22.1,19.3,23.6,18.7,25.4,18.7z"></path></svg></span><span class="essb_network_name">Share</span></a></li><li class="essb_item essb_link_copy nolightbox essb_after_more essb_link_svg_icon"> <a class="nolightbox essb-s-bg-copy essb-s-c-light essb-s-ch-light essb-s-bgh-dark essb-s-bg-network" href="https://getsocialguide.com/robots-txt-file/" onclick="essb.copy_link('1942413643','https://getsocialguide.com/robots-txt-file/'); return false;" rel="noreferrer noopener nofollow" target="_blank" title="Share on Copy Link"><span class="essb_icon essb_icon_copy"><svg aria-hidden="true" class="essb-svg-icon essb-svg-icon-copy" focusable="false" height="32" role="img" viewbox="0 0 32 32" width="32" xmlns="http://www.w3.org/2000/svg"><path d="M25.313 28v-18.688h-14.625v18.688h14.625zM25.313 6.688c1.438 0 2.688 1.188 2.688 2.625v18.688c0 1.438-1.25 2.688-2.688 2.688h-14.625c-1.438 0-2.688-1.25-2.688-2.688v-18.688c0-1.438 1.25-2.625 2.688-2.625h14.625zM21.313 1.313v2.688h-16v18.688h-2.625v-18.688c0-1.438 1.188-2.688 2.625-2.688h16z"></path></svg></span><span class="essb_network_name">Copy Link</span></a></li></ul></div><div class="essb-pinterest-pro-content-marker" style="display: none !important;"></div>
