It is not easy to answer this because you do not give any criteria: religious building?  Sacred place? historical building?

I’ll answer the best I can.

The most important sacred place in Japan in without a doubt Mt. Fuji.
“Mt. Fuji is named for the Buddhist fire goddess Fuchi and is sacred to the Shinto goddess Sengen-Sama, whose shrine is found at the summit. It is the holiest of Japan’s “Three Holy Mountains.” Every summer, thousands of pilgrims and tourists climb to the summit, many of them hiking throughout the night to witness the sunrise from the summit.”  -Source:

http://www.sacred-destinations.com/japan/mount-fuji

One the holiest shrines in Japan is the Shinto shrine at Ise, in Mie prefecture.  It is the imperial shrine and the goddess Amaterasu-no-mikami (goddess of the sun) is enshrined there.

Another very important one is Izumo Taisha in Iwate prefecture.  October is known as the Godless month because all the gods of Japan leave to meet at Izumo Taisha.

In the Buddhist point of view, I would say that the Big Buddha of Nara and its temple, Todai-ji, hold a very important historical significance.  It was build in the first hours after Buddhism was introduced.  The emperor Shomu wanted to both attract the help of Buddha after a series of disaster hit Japan and to assert his power on the provinces by forcing them to contribute to its erection. 

Horyu-ji, in Nara is also very important and is on the UNESCO World Heritage sites list.
The temple’s pagoda is widely acknowledged to be one of the oldest wooden buildings existing in the world, underscoring Hōryū-ji’s place as one of the most celebrated temples in Japan

Now, for each sect, the home temple would represent one of the most important places in Japan.  I wrote a page on Japanese temple layouts using this temple as an example.

As for non-religious historical buildings, I would say that one of the most majestic, by far, is Himeji castle in Hyogo prefecture.  It is currently in renovation (2011) but once it is restored, make sure to visit this beauty of a fortress.

*****
Do you have any suggestions?  Something I forgot?
Please add it in the comments below.

In compassion,

Hugo