Making a developer portfolio readable to search engines and AI tools

Search engines and AI tools do not see your site the way a browser does. They read the HTML, often without your CSS or JavaScript. I spent a day looking at my own portfolio through that lens, and found several things that were invisible to me and visible to them. None of this is a guarantee of better rankings; these are fixes for real problems I could check.

1. Words that only looked separated

My home page headline animates in word by word. Each word sat in its own element, with spacing from a CSS margin. When I stripped the tags to see what a plain text extractor would read, the headline came out as one run-together string like "Ibuildwebandmobileproducts". The fix was a real space character between the words. A margin is a visual effect; a space is content.

2. Answers hidden until a click

Three of my FAQ sections rendered the answer only after you clicked the question. To a crawler that reads the initial HTML, the answers did not exist. I switched them to the native <details> element, which keeps the answer in the HTML and still collapses visually. It also works without JavaScript.

3. Structured data has to match the page

I had a FAQPage block that listed all 19 questions from across the site on every page, even though each page showed only six or seven. Markup should describe what is visible on that page, so each page now declares only its own questions. I checked this automatically: every question and the start of every answer in the markup must appear in the page text.

4. A connected set of entities

Instead of separate, unrelated blocks, the Person, the WebSite and the business are given stable identifiers and referenced from each other, so a page about a blog post points to the same author entity as the home page. Blog posts now carry a real image, ISO dates and an actual word count. Listing pages describe their full list of items. I removed a search action from the markup because the site has no search endpoint, and markup should not promise things that do not exist.

5. Pages with almost nothing on them

I had several blog posts with only a couple of sentences. A page like that can weaken a site rather than help it. I kept pages under 150 words out of the index and out of the sitemap until they had real content.

6. Naming the AI crawlers in robots.txt

Search and assistant crawlers identify themselves with a user-agent name. I listed the common ones explicitly (for example GPTBot, ClaudeBot, PerplexityBot and Google-Extended) so the intent is clear and does not depend on a wildcard. One detail matters: a crawler that matches a named group ignores the wildcard group, so the private paths (/admin, /api) have to be repeated in each named group. If you would rather not allow AI training, you can remove the training crawlers from that list while keeping the search ones.

7. An llms.txt file

llms.txt is a proposal for a short, curated Markdown map of your site for language models. I generate mine from the same data the pages use, so it cannot drift: the services, the pricing, the projects and the articles, each with a one-line description and a link. A longer llms-full.txt includes the text itself. To be honest about it, this is a proposal rather than a settled standard, and I cannot show that any particular assistant reads it today. It costs almost nothing to generate and is easy to remove.

How to check your own site

• curl your page and read the raw HTML. If something is missing there, it is missing for a crawler that does not run JavaScript.
• Strip the tags and read the text. Run-together words and missing answers show up quickly.
• Run Google's Rich Results Test on a page with structured data, and use URL Inspection in Search Console.
• Check that every link in your llms.txt and sitemap returns a normal page.

What none of this replaces

Links from other sites, a complete Google Business Profile, real client reviews and genuinely useful content matter more than any of the above. Structure helps machines understand a page that is already worth reading.

If you want this kind of technical groundwork on your own business website, see my services or contact me.