First, we need to make the distinction between a shrine, a touristic temple and a family-run temple. For the first, please follow the link for an in-depth explanation. For the last two, it is easy to differentiate. First, if there is a small building to pay at the entrance, it is a touristic temple.
Each family has an established family temple and burial ground related to a specific Buddhist sect. Families usually devote themself to that sect for generation but do not even know, in general, what sect it is. Unlike churches, the name of the temple will not be an indication of the affiliation.
The priests of these temples are not really charged with the propagation of the Dharma but are mostly performing funerals and memorial ceremonies when requested.
At the temple, there is a death register stating the real name, posthumous Buddhist name, date of death for all the members of the family.
When going to a Family temple, do not expect to have access to the main hall or any buildings unless you go there for a ceremony. When the gates to the grounds are opened it is to give access to the cemetery attached to the temple.
As far as temple etiquette is concerned, be respectful of the tombs, do not touch anything and remember that they are people's last resting places.
When visiting a large temple, there are several steps you have to take and several unwritten rules to respect. Temple etiquette is mostly common sense but I will state the rules here anyway. As a rule of thumb, do not do anything you would not do in a church, temple or synagogue back in your home country as temple etiquette is very similar to any other religious etiquette.
Make sure you are dressed conservatively. This means no sexy attire for women, it's never appropriate to show off your assets when visiting a holy place. For men, try to wear pants instead of shorts. It is not forbidden but you'll notice that not many Japanese men wear shorts in public. It's always better to try to blend in the crowd.
1-There usually is a booth to pay the entrance fee to the temple. The fee will range from 100 to 500 Japanese Yen, 500 being more common. In exchange for your contribution, you get a ticket and sometimes a brochure explaining the history of the temple. Ask for an English brochure if not handed one, they are often available.
2-Next is the purification pavilion called Chouzuya in Japanese. They are not always present as it is a Shintoist tradition but many temples also house a shrine so it is not uncommon to see one. If there is none, just proceed forward.
If there is a basin, usually covered by a roof, with a dragon or a bamboo pipe sprouting water, use it before going into the temple compound.
3-When it's time to enter a building, first make sure you can go in with your shoes. There is sometimes a shelf with shoes on the side or someone will hand you a plastic bag to carry your shoes inside. This is particularly true when you will exit from another place.
5-Once inside, bow toward the main idol, as a sign of respect. If you are not Buddhist and are afraid of offending your religion while doing so, just realize that paying respect doesn't mean believing in the Buddha. It is a simple sign of respect toward someone who was highly respected.
6- It is better to remove your hat inside. It is not necessary but it is a sign of respect to uncover your head.
7- Some temples are associated with a shrine. It often happens with Shingon and Tendai temples, especially. In this case, there will be some kind of bell with a long rope hanging from the ceiling in front of the altar. When this is the case, you can throw money in the collection box (anything from 10 Yen to 500 Yen will do) and then ring the bell.
After ringing the bell, clap your hands twice, bow once and pray silently.
Clapping is, in Japanese Shinto, the original sound that separated the elements to form the world. It is used to attract the attention of the gods. When a shrine is attached to a temple, it means that the distinction is not really made between a Buddha and a god.
When you are done praying, bow once more, clap your hands once and back away from the place.
7-a-When there is no bell in front of the altar or in front of an idol, do not clap your hands, just join your hands in the gassho position (regular joined hands at chest level used for praying in many cultures) and pray silently.
8-You will often hear some people chanting a sutra, usually the Hannya Shingyo. If you know it, you can join, but it is not necessary.
9-Do not pass between someone who is praying and the altar or idol, no matter how far they are. It is like cutting the connection between them.
10- Do not take pictures with a flash inside the temple and look for a sign indicating that pictures are forbidden as it is usually the case.
11-When leaving the building where an idol is, turn toward it at the door and bow once in respect.
Temple etiquette seems difficult but it is quite easy once you know what you are doing. When in doubt, follow the Japanese. Then again, many of them don't know temple etiquette either. (I had to teach it to my girlfriend.)
Do you have any questions, or corrections on what I explain here? Use this page to join me.