A robots.txt file can easily be created from any text editor. Start by opening a text editor. In this case I’ll be using Notepad.
Provided below is an ordinary setup of how the text file should look. Be sure to include your sitemap URL, user-agents, user-agent allows, and user-agent disallows. To learn more about robots.txt, see my article WordPress Robots.txt.
Sitemap: https://gripptopia.com/sitemap.xml
user-agent:*
Allow: /
Allow: /wp-content/uploads/
Disallow: /wp-User-agent: Mediapartners-Google
Allow: /User-agent: Adsbot-Google
Allow: /User-agent: Googlebot-Image
Allow: /User-agent: Googlebot-Mobile
Allow: /
Once you have the file set to your specifications, save the file as a text document (.txt) and name it robots. Since the file will be uploaded to your website’s server, be sure to save the file to an easily accessible location such as the desktop.
Once the file has been saved, open your favorite FTP client (Filezilla) and connect to your server. Drag your newly created robots.txt file (from your computer) into the public_html on the server. Double click on public_html to confirm the file is in place.
You have successfully added a robots.txt file to your server. Note: You can edit this file at any time. To edit – right-click the file (in Filezilla) and choose View/Edit. The file will open in the default editor. Once you are finished editing, simply hit save and confirm the changed file in the FTP client.