The Gemora in Taanis (16a) states that it was the custom to visit a cemetery on a fast day. One reason given is that the Jewish people were saying that they consider themselves like corpses and this will stimulate them to repent. Another reason is that this will enable the deceased who are buried in the cemetery to pray for them. According to the second reason, they would not visit a cemetery that contained the graves of gentiles.
The Ritva writes that they didn’t go to the cemetery in order to daven there because that is forbidden on the account of “loeg lerosh” – it is considered mocking to the dead who cannot perform the mitzvos; rather they davened in the streets and went to the cemetery afterwards. The Ran adds that they did not take the sefer Torah with them when they went to the cemetery.
The Noda B’yehuda (O”C 2:109) was asked on a year that there was no rain and there was tremendous suffering; if they would be permitted to go to a cemetery with a sefer Torah and daven there for rain.
He cites a Zohar (Acharei Mos) which states that davening by a cemetery inspires the souls of those buried there to inform those that are buried in Chevron (Patriarchs and the Matriarchs) who subsequently will arouse Hashem’s compassion.
However, there is a Gemora in Brochos (18a) which rules that a person should not enter a cemetery with tefillin on his head or read from a sefer Torah in his arm. We can infer from this Gemora that reading from the sefer Torah is forbidden but holding it would be permitted. The Kesef Mishna in Hilchos Sefer Torah (10:6) learns that both are forbidden; reading from the sefer Torah or holding it.
The Noda B’yehuda concludes that although he is not an expert in the hidden portions of Torah, the Zohar cited does warn against bringing a sefer Torah that might be missing letters into a cemetery since this can cause terrible consequences.
The sefer Igra D’taanisa wonders why the Noda B’yehuda makes no mention of the Gemora in Taanis, which would indicate that one can go daven by a cemetery.
The Minchas Elozar discusses the permissibility of people davening by Kever Rochel. Some say that we are not mocking Rochel since she was living before the Torah was given; she was never obligated in mitzvos.
The Netziv rules that in his days, it would be permitted because the custom was to bury them deeper than ten tefachim from the ground and it is considered like a different domain.
The Rama (O”C 581:4) writes that there are places that have the custom to go to cemeteries on Erev Rosh Hashanah and to recite lengthy Tefillos there. The Chidah asks on this Rama from the Ritva in Taanis that states explicitly that one should not daven in the cemetery.
There are those that create a distinction between a compulsory tefillah and a tefillah which is only voluntary.
The Elya Rabbah (581) quotes from the Maharil that one should be careful when going to the graves of Tzadikim that your tefillos should not be directed towards those that are buried there, rather one should daven to Hashem and ask for compassion in the merit of these Tzadikim. Some say that you can ask the dead to be an advocate on your behalf.
The Bach (Y”D 217) rules that it is forbidden to daven to the dead because of the prohibition of being “doresh el hameisim.” He points out that even though we find that Calev did daven in Chevron by the Meoras Hamachpeila, he wasn't davening to the Avos. Rather, since a cemetery is a place of holiness and purity, the tefillos davened there will be more readily accepted.