Ghost DNS Lookup Puzzle on NIS Clients

I faced an extremely weird problem of ghost DNS lookups on machines configured as NFS clients. It took quite an effort to solve it. Here are the details.

Configuration
I have a machine configured as NFS & NIS server, named database. There is another machine (named internet) configured as caching DNS server and also to provide DNS resolution for local machines. All NIS (& NFS) clients have been configured with static IP and DNS server set in resolv.conf

Problem
On the machines configured as NIS client, I could resolve host names properly with the DNS servers specified in resolv.conf. Unfortunately I couldn’t resolve the hostname properly for database (NFS & NIS server) machine. It always resolved to an apparently harcoded IP address of guide.opendns.com. I restarted the NIS server as well as the client machines, without any benefit. I also searched for the IP address as well as the hostname in both the server and the client machine without success.

The second issue was even more bizarre. The NIS client machines were properly resolving hostnames like yahoo.com or google.com even when the nameserver was commented out in resolv.conf! I ensured that the names were not in cache or in hosts file.

Explanation & Solution
The NIS clients can resolve names through NIS server. I couldn’t find this documented anywhere. I derived at this conclusion by experimentation. So even if your DNS server is commented out in resolv.conf, it still can resolve using NIS server.

Secondly the NIS server caches the IP address of itself along with the hostname. This cached information gets preference in NIS clients even over DNS lookups.

The solution is to the following (again) on NIS server to regenerate the cache:
/usr/lib/yp/ypinit -m

PS. In the next article I will talk about a BIG security hole in NIS which has persisted over several years. Finally I will disclose it.