The oldest example of Western architecture in Matsue, on the West coast of Japan, is the Dutch Mansion. The Made in Matsue blog explains (pictures after the link):
This building known once as the “oranda-yashiki”(ã€Œã‚ªãƒ©ãƒ³ãƒ€å±‹æ•·ã€), or Dutch Residence/Mansion, is said to be the first example of western architecture here in the Sanin Region when construction was completed in 1871 (4th year of Meiji).
Originally built to serve as a temple school, ‘Omachi School’ (è‹§ç”ºå¦æ ¡), it is quite extraordinary that it has remained intact in its original form to this day, as buildings of this style have almost all been reconstructed on or after 100 years since construction.
It is not clear from the article whether this building was actually based on Dutch architecture, I am guessing that ‘Dutch’ in this case may have been shorthand for ‘European’.
Between 1633 and 1853 Japan closed itself off from the rest of the world, a policy called Sakoku. During this time only a limited number of countries were allowed to trade with Japan, the Netherlands being the only European trade partner. Dutch traders were confined to an artificial island in the harbour of Nagasaki called Dejima.