In my previous report, the timing difference is by a factor of 45. That's because i had stuff in my init file. (i have hooks for both find-file and html-mode, and that probably caused the major slowdown.) Now, the factor is actually just about 5.
You should still use with-temp-buffer instead of find-file besides speed, because find-file has quite some side effects.
Big thanks to Trey Jackson.