Found it!

For now I'll name the ugliest building in Toronto to be Robarts Library.
Perhaps one day this brutalist building will seen in a different light, for now it can be valued for its expression of late modern architecture, campus building design and exposed concrete details. Simply put though its an eyesore.
Robarts is the big central social sciences library of UofT. It was the trend in campus buillding in the 1960s to build these exposed concrete buildings like this, starting with the Yale Art and Architecture Building. Unfortunatly they are very un-human.  

(Picture from Wiki)

From the street, you can't really appreciate the whole form, which is a giant triangle. Then inside its quite disorienting and there is very little natural light. To get to the bathrooms in the service core (in the center) is like going to the Mines of Moria.

Most surprising is how it looks like a prison complex and dominates the Northern Vista. This was the view from upper level of the Art Gallery of Ontario. 

Its equally interesting that Chicago has a very similar triangular brutalist building in the downtown sector.  (Pics here from the Web too)

However in Chicago their building is a PRISON, not a campus LIBRARY. And the small windows are to keep the inmates from not escaping.
We, in this household have dubbed the Robarts the Robarfs.

More about brutalism here.