In a React.js app, you can add a robots.txt file to control how search engines crawl and index your site. Here’s how you can create and include a robots.txt file in your React.js application:

1. Create the robots.txt File:

Create a robots.txt file in the public directory of your React.js project. The public directory is where you can place static assets that will be served directly by the web server.

The structure of your project might look like this:

2. Add Content to robots.txt:

Open the robots.txt file and add the content according to your needs. Here’s an example of what you might include:

User-agent: *

Allow: /static/

Allow: /public/

Allow: /static/js/

Allow: /static/css/

Disallow: /admin/

Disallow: /login/

Sitemap: https://your-site-name.com/sitemap.xml