I find it fast enough for reading, but not fast enough for skimming or navigation. If I want to page through a book quickly to find something, it just lags way too much rendering the pages; I have to stop every 5 or 10 pages and wait for it to fill them in, or I'll be paging through nothing but whitespace. But it's fine except for that.
Keep in mind that PDF.js is an engine + a viewer. Just a fast engine does not make a nice experience. There are many things in the viewer that one can do to make the experience of using it feel a lot faster.