Research Dashboard
Agent Compliance with SOM Directives
Six experiments testing whether AI agent frameworks discover and use the proposed SOM robots.txt directives. The server works. The parsers work. The frameworks do not.
0 / 5
frameworks compliant
check robots.txt
0 / 202
server tests pass
0 / 3
parsers compatible
0 / 12
negotiation correct
The Discovery Gap
Server infrastructure100%
Parser compatibility100%
SOM accuracy vs HTML100%
Framework discovery0%
Everything works except the one thing that matters: no framework looks for SOM. The entire discovery chain fails at step zero. This requires ~50 lines of code per framework to fix.
Key Findings
01All tested robots.txt parsers (3 libraries across JS and Python) parse SOM-extended robots.txt without errors — confirming RFC 9309 backward compatibility.
02No parser natively extracts SOM directives — they silently ignore unknown directives as RFC 9309 Section 2.2.4 requires.
03A simple regex pre-scan (15 lines of code) successfully extracts all 5 SOM directives from any robots.txt.
040 of 5 tested frameworks check robots.txt before fetching pages.
050 of 5 frameworks send Accept: application/som+json — all receive HTML by default.
06BeautifulSoup-based extraction CAN discover SOM meta tags in HTML, but no framework proactively looks for them.