Alexandra hall was build as public swimming baths in 1863; apparently the original swimming bath tiles can still be seen in the basements. It became a public meeting hall sometime after this.

The exclusive section of the brethren met here for some time before moving to their own hall on Bennett Park. (Edwin Cross: Kelly, p45). 

"Dr ROBERT M’KILLIAM, in 1880 succeeded to the practice of the late Dr. Tate, of Blackheath. In a short time an Assembly of believers was formed at Blackheath. The Alexandra Hall was taken, and up to the time of his last illness the doctor continued to preach the Gospel and minister the Word there, and very many will have cause to thank God for his faithful ministry. His activities were not confined to Blackheath, as he travelled much all over the country and laboured for the Lord most assiduously. He was Editor of The Morning Star from its commencement in January, 1894, and continued so till the time of his death—thus covering just over 21 years. Through this paper, which was “a herald of the Coming of the Lord Jesus Christ,” his name was known and his teaching appreciated and blessed in many parts of the world." (Chief Men Among the Brethren)

It was listed in the open brethren meeting books in the 1890's, and also listed as such in 'Religious lfe of London'.

"Early in 1906 we moved to Blackheath, a suburb in southeast London. ... About a quarter of a mile from where we lived in Blackheath was Alexandra Hall.  In some ways the assembly there was different from others.  Most of the members were upper-class people and only a few of the poorer class.  There was one woman who was a charwoman but rather outspoken and uncouth when she poured out her scorn on the snobbery of some ladies there.  Mother’s ministry was to befriend some of the old ladies and I was not enraptured over such guests!  She also took an interest in young women in domestic service who were away from home.

When we first moved there, the congregation at Alexandra Hall was quite large.  Some two to three hundred would meet each Sunday morning to observe the Lord’s Supper.  Without fail one brother, Mr. Luck, would read a chapter from the Bible without comment and Easter Sunday it would surely be Luke 24.  There would be some five hundred at the evening service.  Dr. Robert McKilliam was the recognized leader in the assembly and always preached when he was home.  Though a medical man, he was a great Bible student and also editor of a monthly magazine, “The Morning Star,” which emphasized the Second Coming.  The choir, mostly of older persons, sat on the platform.  One gentleman had his favorite seat by the railing and during the sermon he would lean on this railing and enjoy a little sleep.

The assembly regularly held a brief 15-minute open-air service on the steps before the evening service, summer and winter.  Then in the summer there was an open-air meeting in the center of the town after the service.  Later when numbers were dwindling they held an evangelistic campaign.  The invited preacher was an eccentric man by the name of Hodson.  His zeal for the Lord was accompanied by a sense of humor and a quick repartee.  On one occasion, so I am told, he carried on a tent campaign in an English village.  Sunday morning he went to the Anglican Church to hear the parson.  Being sadly disappointed by the sermon he asked the parson, “Do you call that preaching the Gospel?”  It seems that in retaliation the reverend gentleman persuaded two village thugs to beat up Mr. Hodson and burn down his tent.  When they accosted Mr. Hodson in front of his tent and stated their intention, he, being a big husky man, picked them up one by one and tossed them over the hedge.  The parson who had been standing at a distance, approached and haughtily asked, “Mr. Hodson, do you call that preaching the Gospel?”  To which Mr. Hodson replied, “No, I call that casting out demons!”

Perhaps the dwindling attendance was in part due to the fact that there was no Sunday school and very few young people at Alexandra Hall.  After we left there Dr. McKilliam passed away which further contributed to the decline.  During my war service in World War I, visiting Blackheath one evening I found about twenty believers meeting in a much smaller hall."

Cyril Brooks, Grace Triumphant, http://www.plymouthbrethren.org/article/899

Evidently it fell out of use, and it was sold for use a as a cinema in 1921 but never actualy used and was then sold in 1928 to Lloyds Bank.